458 karma · joined February 20, 2018
https://diggerdev.notion.site/DX-platforms-for-your-cloud-ac...
The idea is to facilitate ci/cd for Terraform without a dedicated ci/cd stack. Tools like Atlantis or Spacelift are effectively alternative CI systems with compute, orchestration, UI for logs etc. Why not reuse the infrastructure of your existing CI system for that? So that terraform binary runs in managed compute just like any other ci/cd job. To achieve that we've built a thin wrapper atop terraform that reads config from your repo and manages locks in your cloud account. This way you don't need any manual trickery to set up ci/cd for terraform; it's just a drop-in Github Action.
Also a list of startups building similar value propositions - some great products there: https://diggerdev.notion.site/DX-platforms-for-your-cloud-ac...
We however realised recently that this is not something that can be tackled in one go by a single product. Because this is multiple products solving multiple problems in one package.
We still believe that this "end state" of an OS-like layer that'd manage cloud-native services like hardware, using IaC as an "assembly language" of sorts, is inevitable. It will get built one way or another. But not as a platform from scratch; rather, as an expansion of something much smaller.
What could it be? If the "assembly language" hunch is right, then it must have something to do with IaC. We played with an idea of a tool that'd generate Terraform based on higher-level concepts as inputs. Here's one (UI): https://terrabook.io/ And another (CLI / framework): https://diggerhq.gitbook.io/trowel-docs/getting-started/what...
The framework idea still looks promising, we haven't fully explored it yet. But even with a perfect framework in place that "compiles" into Terraform, you'd still need something to take the output and make sure the changes are reflected in the target cloud account. Continuing the "assembly language" parallel, think bytecode, the missing piece of the puzzle appears to be a "runtime" for IaC.
So we are building such a runtime. Funny enough, it strongly resembles CI - you push Terraform code to Git and it gets deployed. Sort of an "orthogonal" CI pipeline to your normal application build-deploy CI/CD.
But existing CI tools are ill-suited for Terraform, because it has state and that creates potential for race conditions; while most CI tools assume stateless jobs. A class of CI/CD products for Terraform exists (Spacelift, Terraform Cloud, Atlantis) but they are more like separate full-stack CI systems. We think that having 2 CI systems for that doesn't make sense. The infrastructure of asynchronous jobs, logs etc can and should be reused. Stretching the "assembly language" parallel, this is a bit like the CPU for a yet-to-be-created "cloud PC".
So it boils down to making it possible to run Terraform in existing CI systems. This is what Digger does - check it out: https://digger.dev
It took us 2 years to realise one simple thing: it's not wrong, it's just not the right starting point.
Indeed this kind of platform that'd make cloud ops much easier, decouple dev from ops, self-service and everything, on top of Terraform, is inevitable. It makes perfect sense!
But people don't need platforms. People need great solutions to at-hand problems. And for such a platform to take off, it needs to be best in class at _every single thing_ it does - everything from infrastructure provisioning to ci / cd to observability. No team in the world has capacity to execute at such scale from scratch, not even Microsoft (owning both Github and Azure).
Worse still, all these problems occur at different times in customer's company lifecycle, and often to different people in their org structure. This makes it next to impossible to narrow down to a use case or specific customer persona. This feels like a force that constantly tries to tear your startup apart. I'm sure you know what I'm talking about.
I think that the only way to make a horizontal platform is incremental expansion of independently successful vertical parts. Just having "integrations" with various third parties is not enough - for every problem your platform solving itself, there's likely a specialised tool that does it better. So there's got to be _one thing_ that you do better than everyone else. Just one. Unless you have it, people won't want your platform. And when you have it, you might just as well not have anything else because none of the other stuff would matter.
Find the one thing that people struggle with, build a simple solution, and make sure you are the best in the world in this one little thing. That's not hard to do, but extremely hard to truly believe in deep down. But that's the only way.
We have recently found our _one thing_: CI for Terraform. Digger makes it easy to run terraform in Github Actions. That's it, that's all we do.
Good news: this does not invalidate the big platform vision in the future - you can still build up towards it incrementally from your small thing! If anything having tried that is a competitive advantage against clueless incumbents.
I also may be wrong. Prior to Digger I was working on Palantir Foundry which faced a similar "backwards GTM" challenge - so I have 2 biased data points. But the folks at Qovery seem to be doubling down on the platform route. I'm rooting for them as well as for everyone else on the list. With enough perseverance and capital it might well be possible to pull it off, especially via top-down sales-led enterprise route. Someone will eventually make such a platform because it's inevitable.
We strongly believe that some kind of a layer above present-day IaC is missing and badly needed. What exactly is the right solution isn't 100% clear just yet; but it is along the lines of a generator / framework / compiler / "OS" that would provide reasonable defaults without taking away flexibility. Then there's also a challenge of one-off usage; generator alone is only useful once; a framework on the other hand is helpful all the way throughout the project's lifecycle.
Terraform looks like the best (only?) "underlying" IaC, more like a target "assembly language" using the LLVM parallel. The crux seems to be the runtime - it is less about any given state of infrastructure (that's already captured pretty well with existing IaC tools) and way more about the logic of transitions between those states.
Managing state / running it on the server (the "CI for infrastructure" use case eg TF Cloud / Spacelift / Env0 etc) seems to be the logical extension of the baseline case of initial provisioning. We started with this, but now are viewing it as secondary consideration that will fall into place once the baseline framework scenario is figured.
We Keep iterating - and Lemon 2.0 is one that makes us most proud of our creations to date. We launched the original Lemon 13 months ago. Since then we rebuilt the UI from scratch, added a bunch of cool features and most importantly - killed a bunch of "meh" features.
Its now centered around your apps. No more confusion with environments and services. Simply apps.
Lemon no longer requires your project to have a Dockerfile. We use one if you have it, but if not we try to detect the type of your project automatically and generate an image on the fly. Si you can deploy drontends, backends, whatever. We also support monorepos.
Finally we added custom Terraform editing in the UI. So you can simply drop whatever TF you have and Lemon will run it.
Would love to hear your feedback!
Igor here - the founder of Movicorn (by Digger). Movicorn helps move to AWS from other cloud provider in one click.
We came up with the idea for Movicorn after a lot of our users at Digger kept asking us for migration to AWS from their existing cloud providers. The reasons they gave for moving to AWS ranged from having AWS credits, AWS offering better security and compliance, and better geographical coverage by AWS servers.
We broke the process down into just 3 steps:
1. Connect your GitHub 2. Connect your AWS account 3. AWS deployment using Digger (it generates and runs Terraform)
We think this can reduce DevOps and infra overhead, instead of spending weeks on migrations you can go live in minutes and gradually customise.
Please let us know what you think, your feedback is extremely important to us!
my thinking with Terrabook was like, what stops non-devops-savvy developers from starting to use TF? And the hypothesis is that it's a bunch of things that need to be set up prior to getting to something working, like a VPC and so on. And then having that, how do I get a database? The fastest way seems to be just find someone's github and copy from there, assuming I'm not familiar with the TF world. So I thought, why not build this right into the tool
With Terrabook our hunch is that Terraform can be made much easier if it's generated in small chunks. The language itself is trivial, lightweight json basically. Nothing to learn there. But the AWS part is hard. What are the options? What should you write? And then having spoken to many users from previous launches we learned that people google TF snippets and copy-pasting them and tweaking just a bit. It's much easier to change if the bulk is already there. So we made this snipped generation pice with a UI inspired by Jupyter / Google Colab / Replit to basically lower the entry barrier into Terraform.
Today’s launch was born out of an idea from one such feedback point - AWS cost alerts are delayed (sometimes for up to 24 hours) and cause nightmares for founders. We understand this and have faced this problem ourselves. That’s why we decided to ship CostGuard, an easy way to get alerted when your AWS costs skyrocket. So that you can focus on building and not be unnecessarily worried about AWS costs.
This is how it works:
1 - Connect Your AWS Account
2 - Deploy your stack on AWS with Digger (it allows us to track the infrastructure changes)
3 - Get alerts in your email inbox when your costs increase significantly
Would be super grateful for your feedback in the comments, this is what gives us direction and a roadmap to building a better DX at Digger. Thanks.
StackBricks allows you to start building your app with a pre-made stack that works right away in your AWS account. A bit like create-react-app but for your entire stack. So instead of building your app first, then figuring out infra it is live from the moment you start working on your app. All you need to do is replace dummy apps with your code by connecting your GitHub repositories.
You might have noticed a bunch of things from us - the original Digger last year; AWS Bootstrap, Terragen and OpsFlow more recently. They’re all powered by Digger engine that automatically generates and runs Terraform. On top of this we have built multiple versions of the UI that we keep changing rapidly based on your feedback. With every launch we learn what people like and what doesn’t fly and make the product better.
The idea behind StackBricks is to combine the features that got the users of AWS Bootstrap and Terragen most excited. While AWS Bootstrap only allowed you to set up a “triangle” stack (frontend + backend + postgres) and Terragen was all about auto-generating Terraform, StackBricks introduces the concept of generic “stack layouts” (you can see them right on the landing page). And from Terragen we have taken and improved on the idea of overrides - you can write your own custom Terraform right in the web UI, it no longer requires you to connect a dedicated infrastructure repository, which was a common friction point before.
We've also just launched on ProductHunt: https://www.producthunt.com/posts/stackbricks
Please give it a try - would love to hear your feedback!
We went back to the drawing board and built BotDeploy to help engineers deploy their bot to AWS in three simple steps.
1. Select from Slack, Discord or Telegram 2. Login with GitHub 3. Deploy to AWS!
We genuinely believe that we have cut build time from days to minutes, and would be super grateful for feedback from the HN community.
We then launched a number of more focused products, all powered by the same Digger engine. 2 did better than the other 5 or so. Lemon was an alternative UI for AWS; Alicorn was a multi-cloud offering for containers. Those launches helped us realise that some of the core architectural assumptions we had were wrong. We also needed to rework the UX because people were getting confused by the split between Services and Environments, as well as separation of Infrastructure and Software deployments.
So we got back to the drawing board. We have radically simplified the UX, removed the confusing parts, introduced keyless AWS connection with narrowed down permission scope and many other things. The question remained - which is the use case people care about most? To answer it we started launching smaller products again, all powered by Digger engine with tweaked configurations.
Many people liked AWS Bootstrap. It allows you to quickly configure your AWS account to run frontend, backend and a database of your choice. Quite literally bootstrap.
Another thing that was well received was Terragen. We made it all about auto-generation of terraform. As soon as the user connects their AWS account they can export generated Terraform into their GitHub.
With OpsFlow we took the learnings from AWS Bootstrap and Terragen and made the UI even simpler. It no longer bothers the user with optional stuff, it is all moved to the new Settings page. And it's centered around 2 simple types of building blocks - Apps and Resources.
OpsFlow is the closest we got so far to making something people want. Still a long way to go though :)