Taking on Good Technical Debt
matt-rickard.com
matt-rickard.com
Somewhere along the way, it became fashionable to build out your devops on day one as though you’ll be Google tomorrow, but to write your code without any architectural foresight as though the only thing that matters is tomorrow’s deployment.
It’s always nice to see people advocating towards a thoughtful balance.
To build the servers I was running on I wrote an Ansible job to install everything and copy the code down. All I had to do was add a couple flags to that to allow it to just deploy new changes. Instead of getting into the CICD pipeline I just took a few minute to make a little web front end for myself to kick off the job and watch the job status (we have Tower, but it's painfully slow). I can now deploy in 18 seconds with 1 click. Seems good enough. I'm sure I could set up a web hook to eliminate the click, but it's not a big deal.
I try to align when possible. I think your solution is hacky and costs the company much more in the long run.
I used to try to align when possible, but lately it hasn't made a lot of sense. Keeping up with the tooling ends up being the whole job, as they change it every 6 months. This site is a side project to help out the team, not my primary job. I can't be dropping everything each time they decide to switch vendors, change up their variables names, or whatever else they decide to do.
At my company, it would involve going to weekly meetings to get all the updates and changes happening as they keep developing it, one-off meetings to get onboarded and setup... I'm pretty sure they also designed the system for Kubernetes, so I figured it would also mean containerizing the site, as it has been running on VMs for almost 10 years. Those weekly meetings also mean constant changes so things keep working (which I'm not dealing with on a different site that is going though CICD). All of this in the midst of a lot of bureaucracy. It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.
It likely would have taken several months for little to no tangible benefit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited. It took 8 months for someone to bother to tell me where I could see build status/errors, at first they just told me to "wait about 10 minutes". There is no documentation anywhere. It's all tribal knowledge and no one seems to have the time to share it.
The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.
Yeah - this is key. If you're already set up to _not_ be containerised, then it's likely not going to be worthwhile unless you are actively feeling some of the pain points around running on VMs.
> It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.
One saving grace is that a huge amount of what k8s does is abstracts the aws-ness from underneath. Running a live application on k8s on aws is three separate skills - running a live app, running apps on k8s and running k8s on aws. Two of three of those still transfer!
> fit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited.
That's fair, and it's up to you to make the call as to whether it's worth it or not. If your team aren't willing then the use cases are very limited, and that's a political fight!
> The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.
I've been a driver for improving developer best practices on every team I've worked on. Not everything has always stuck, but I've found that processes that can be automated tend to have high rates of success, and processes that rely on humans tend to be sidestepped. I spoke to a coworker from a previous job last night who told me that some of the process fixes I made as a cog in the wheel have survived 2 years without me there. All is not lost!
Yeah, it's just a matter of trying to pick those horses that seem likely to stick around, or that actually make like easier. Ansible is one of those; the CICD process and everything around it is not. We've changed out container platform 3 times in 3 years, and just kicked off a project to change it a 4th time, while also opening up some public cloud access if it can be justified. It's chaos.
There was also a push to build a lot of home-grown tools over the last 3-5 years. All of that is being ripped out in favor of vendor provided solutions.
The primary things that seem to stick around are the snowflakes, as sad as that is. The only teams that don't seem stressed by all that change are the ones who just do their own thing. I fought the good fight, jumping the latest and greatest stuff for over a decade and trying to put systems in place to get others to buy in as well, but I'm tired and this latest crop of tools seemed like they were destine for failure from day one.
Sounds like your system is way more efficient than the CICD... in the short term, provided you're the only person that ever needs to use it (and/or nothing ever changes)
This seems like a very large assumption.
> as they know Ansible, but have no idea how the CICD pipelines work
I know Ansible very well. I have never set up a CICD system and would much prefer to write a bunch of Ansible playbooks for myself than to do so. My company has a CICD system already set up (I have no idea how it works internally). Setting up a pipeline and understanding how that pipeline works as a user of the CICD system is still a lot easier for me than figuring out someone else's Ansible playbook repo. Adding jobs to that pipeline is still easier than writing new Ansible tasks. CICD is almost always easier for dev teams regardless of their experience once the initial complexity of CICD setup/hosting/maintenance is already handled.
If you don't already have a CICD set up, then setting up one is likely overkill for small teams. If it's already there (and maintained by a well-resourced team), use it, unless the CICD platform being used is absurdly bad for some reason (I've used a few and they have been largely fine / interchangeable).
As mentioned in another comment, the site has been running on VMs for almost 10 years and not containerized. The CICD pipeline is built around Kubernetes and I have no real reason to move to Kubernetes other than to say it's running in containers. The VMs have never had issues, it's a relatively small internal site, so I don't need to worry about scaling issues. K8 is also in the process of being replaced, so moving to it now seems silly. I use the pipeline for another project I'm working on with another team and while it's needed for the stack they're using, it's bad. No one seems to know how it works, I haven't spoke to a single developer who has a positive thing to say about it, it's very poorly documented, and any time there is an issue it requires messaging a bunch of random people until you find someone with the right blend of tribal knowledge and time to help. I've had a half dozen people come to me just to find out where to see the deployment status or to know how to promote something to upper environments... and I wasn't given that information until 8 months in. Seems like it would be day 1 stuff that would be easily documented.
For a hackathon they made something to do a one-click setup of a project that I would likely use, but outside of the hackathon, that feature isn't available and it would require months of work when factoring in meetings, waiting on other teams, etc to get setup without that. It's just not worth it for my use case.
But the calculus changes if instead of EC2 nodes you can have a helm chart that deploys directly to a managed kubernetes services like EKS or GKE?
If you have a lot of experience with any interface, you can do it really efficiently, but if you don't, kubernetes can make your experience worse than anything you'd come up with on a cloud provider.
Then you'll need non trivial amount of knowledge of how it works to avoid many pitfalls (the helm chart will probably only go so far, it will still need small tweaks for your specific use case)
So it's a combination of upfront time investment, ongoing tuning, and recurring maintenance events.
That's a big part of many k8s pitches I've seen but in my experience it hasn't been representative of reality (or at least for every part of infrastructure you no longer need to run there's a new part of k8s architecture/config to learn & manage).
For me the selling points are:
1. For the time-cost of upfront complexity and setup, you get extremely easy and quick subsequent addition of infra/tooling & scaling thereof. 2. There are certain patterns of tooling that k8s enables (relatively easily), that would have been difficult/impossible with more traditional setups.
I don't think it makes infra management overhead less, it just reorganises it so it's more upfront and then less later while scaling.
Even with ECS which is way simpler than K8 for small teams using Fargate has been incredibly beneficial in terms of debt/time.
Then next step would be to move the entire application onto Lambda so we're not paying for downtime, but that's a whole headache and a half.
Whatever claims about user friendliness, I didn't figure a stable way to do deploy separate docker-compose.yml's from same user at the same time.
> Coming from AWS to Linode VPS with Caddy + Docker felt great
I guess that depends what you were using with AWS, but Fargate + k8s is a pretty amazing workflow.
> Im wondering at what point (N requests per second ?) it becomes tech dept
I make these calls for our org, and I wouldn't call it tech debt based on the performance or scale, but because of maintenance. As an example, if you needed to hire a new person/replacement person for your team, would it be easier to find someone who can maintain k8s or docker compose?
The other area it shines for is the ecosystem. If you need to add monitoring to your app (e.g. with Prometheus), adding it to a k8s cluster is `helm install prometheus`
My last team was the same, there was ~100 developers on the main app, ~50 on the ancillary services, and a ballpark of ~10 on the "platform". I was just a team member on that team. Current org is 15 devs, 30 people total.
> My concern is that there is a cost to magic of adding deps like that.
Of course there is, there's such thing as a free lunch. It's up to you to decide which is right for your team/org. The cost of doing it manually is developer time, tech debt and the bus factor. The cost of using helm is introducing an abstraction. Profesionally, the bus factor and developer time of home rolling a solution make relying on existing solutions a very appealing proposition. Having to maintain a set of half baked shell scripts is exactly what tools exist to solve, and I've had more issues with deployments where someone has bunged together a bash script that makes varios assumptions about tooling, versions, installation paths, working directories, path formats, etc. than I have with problems related to our actual deployment infrastructure.
> I guess my question is at what org scale the tradeoffs are right
The cost of switching is huge, and likely not worth it unless you have genuine problems that would be solved by moving. For a greenfield project, frankly I'd think you're wrong to use docker-compose on any new project. Either you're managing the underlying infrastructure (ec2/vps style) or you're not (fargate/DO App Platform style), and in either case docker compose requires roughly the same amount of investment for a smaller return on a less actively developed product objectively solves less problems and requires more work to integrate.
[0] https://github.com/marketplace/actions/deploy-to-kubernetes-...
Personally, I would run k8s from the day I expect to run the code as separate deployment to running it from my IDE. I have the skills to do so, and I found I lose way less time doing so than trying to do "classic" server setup, even if it's just one server.
Everything else would only mean less golden path support, less synergies etc.
As someone who works in a small team within a large corp where we use k8s for everything large and small, I did not read that statement as applying to me.
If you have a subsystem conform to a good API and is well behaved it can be rewritten. It is allowed to have some tech debt.
If the subsystem is sprawled through other subsystems that tech debt doesn't get paid down because now you need an expert in all those areas to fix it.
I swear I have worked places where there is code no one currently at the company can understand. At best you micro-fix for bugs and back the hell away!
It's funny and gross enough to stick most of the time.
That said sometimes abstractions leak. I was using Firebase and getting the public Url for a blob given it’s path is an async call! God knows why.
What I see is that the term makes technology look bad, doesn't work as a lever and demotivates developers.
Be a craftman, don't do technical debt. For time to market create a zerocode/crappy MVP that you throw away. Schedule real technical debt like DB migrations and do them as needed.
> For time to market create a zerocode/crappy MVP that you throw away.
Unfortunately what tends to happen is that as the business sees early success and starts to grow, there is never a stopping point where people are happy to start again and build from scratch 'properly'. That crappy MVP will be extended, adjusted and corrected, until it's something no longer throwaway but also not well built/tested/hosted/monitored/etc.
Maybe slow down to go faster is a better approach.
Yes and from talking to my coachees, this is one of their main problems. I urge them to throw away the MVP when they got funded.
If business requirements change, that isn’t tech debt.
If you don’t implement new high-level (business) insights or postpone this work, you are accruing technical debt. At least, that’s what Ward Cunningham stated.
That's the opposite of what I wrote. My scenario was you knew that it didn't make sense for that project but you still did it because of time/budget/resource reasons. You start paying off the day you can't scale it anymore.
If you make a bad choice with no valid reason then it's just a bad choice, not technical debt.
Nobody built the pyramids in a day.
Here is a different attempt on definition: An occurrence on debt is when a higher up decides that a product/construction must be ready when it clearly is not, and thus convinces/bullies the engineers and put their signature on it.
Analogies can be a great tool to communicate or for prose but don't take them literally.
Unfortunately your description is what has become the common usage of TD, and now the term is meaningless.
Frankly speaking, today, TD is just a neutral term to externalize blame/responsibility.
Or clean code because no one will ever touch it again.
Or a feature you only need once and just write it somehow.
And then all this shit stays in your product and starts dragging you down.
This is totally different than when you start with a monolith and know that you might want to rework it later but accept potential future work.
And you should start with a modular monolith anyway.
I don't think it's reasonable to try to redefine the term technical debt.