Diagram as Code for prototyping cloud system architectures | Hacker News Reader

Diagram as Code for prototyping cloud system architectures

github.com

129 pointssimplecto30 comments

The entire contemporary devops process is awfully convoluted. In fact, I've heard DevOps people call themselves "yaml engineers". Often in your docker compose there is some stupid error, there are 3 versions of Docker compose files, there are 7 ways to do something, Docker CLI manual can be printed in 1000 pages and we've created extremely bloated (despite of caching whatever) CI/CD. For what? Dependencies? Let's download 4.5 GB of images because my app needs to be reproduceable.

I tried reading about Kubernetes. Holyshit what a monster we've created. I am sure there is going to be an orchestrator abstraction for multiple Kubernetes clusters. And then we have environment variables, secrets, access keys, a bazillion different parameters that need to trickle down from Gitlab CI settings all the way to the app through many layers.

Good lord. All engineers should be ashamed of this mess. We've forgotten how to build simple, beautiful, lean and parsimonious software. Instead of writing logic, I am configurating things all day.

Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers. But, let's shit on the F-35 program everytime there is a Lockheed article on HN. Because surely, software engineers are the smart bunch ya. I know, I am unreasonably obtuse here, but jeez, no one complains about today's deployment process,... its apparently accepted without a question.

Multi-zone reproducible HA environments are complex. Who'da thunk it?

Multi-zone?

Geographically disparate datacenters

Kubernetes and Docker is about moving complexity, not removing complexity. I recently held a presentation for our UK applications team (manager, team lead) about our setup from a provisioning perspective, release process perspective, organizational perspective, QA perspective, operations perspective, troubleshooting perspective, full traceability, ease-of-deployment, time-to-market.

Needless to say they were amazed to see that we can deploy a new isolated, "virtual machine" application running in seconds using dynamic DNS, rather than waiting a year for a new application server from the IT organization not to speak about deploying it, operating it and troubleshooting it.

> Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers

I'm with you on the feeling that modern stack is bloated and heavy... However, this is exactly the scenario where the compiler bug is isolated quickly because you're not chasing dependency differences between boxes

No! What if there is a bug in Kubernetes itself? or in the Docker engine? Or a compiler bug that affects Kubernetes in mysterious ways?

What if there is a bug in kubernetes or docker?

There are tons of them, people find new ones all the time. You troubleshoot it like every other problem.

Hell, you’ve barely scratched the surface of all the ways a kubernetes install can break — coredns? Etcd? Nginx? Istio?

My god what if there’s a bug in the Linux kernel? We should just give up, computers were a mistake.

With all due respect this argument in itself simply is not valid. The engine itself runs, but you WILL need a driver.

I started doing diagrams from code and don't want to go back to dragging things with a mouse again. VSCode plantuml gives live feedback. I started using this for c4 diagrams https://github.com/RicardoNiepel/C4-PlantUML

That's really cool to have the charts generated as code. Very concise way to specify things, and with all the power of graphviz behind the scenes.

Another tool in that space is https://cloudcraft.co/ although only for AWS, and you have to manually do the diagram layout.

I've used asciidoctor with it's diagram plugin (https://asciidoctor.org/docs/asciidoctor-diagram/) internally on memos/draft documentation, which that gives us an ability to actually just check in the asciidoc with diagram source into git. Then we use merge requests for a real review process. Then, I just usually just export to pdf or word for broader sharing among the masses. (Amazing how this usually gets far more attention than Confluence, etc)

Main benefit is that most of these kinds of CLI diagram tools have much longer lives than any GUI-based diagram software I've used.

But even just having a separate "diagram source" file where you just check in an image is still nice. Then you don't have to bother with wondering who has access to the tools, etc.

This is a great idea. I'm not sure about the other major cloud providers, but AWS has the AWS CDK which allows you to use TypeScript, Python, Java, and .NET to create your cloud infrastructure via code. The ability to generate cloud diagrams from AWS CDK code would definitely would be awesome. Hopefully, your work gains transaction, then I can envision other devs writing adapter code for your project.

I have a script that we call when generating cloudformation templates to generate a dot graph

Cfn2dot source https://github.com/tobiipro/support-firecloud/blob/master/bi...

Used like this https://github.com/tobiipro/support-firecloud/blob/master/re...

One problem is that you need meta-data to give weights to different resources so that you highlight a lambda resource and not there associated cloudwatch loggroup resource, and its iam role, and iam policy, and its permission resource, etc

The other problem is the graph itself shows how the cloudformation resources depend on one another, not how your data/execution pipeline flows.

So you need to do it manually or maybe try to add lots of meta-data to the types of resources.

I wish it was an extension to plantuml instead.

Finally related topic! I've been meaning to ask, is there any tool (or a suite of tools) that can be used to visualize network along with real-time health status of each nodes? Imagine you have a web server, API server and DB server, lined up left to right, each connected with a line. Each of them has green dot, and when they're down, the dot changes color to red.

The diagram can be manually laid out, but the "health indicator" must be updated automatically.

We have customers doing this with yFiles. But that's a generic graph drawing and layout library, so getting from your network topology to a graph structure and getting health events is not something the library helps with. You'd get automatic layout, however, and colored dots are on the trivial side for a visualization.

`kiali` maybe the thing you want. But it should be used with Istio stuffs.

DOT notation is simple, and works great for generating diagrams.

Another one if you like dot-esque notation and pretty output out the gate is https://mermaid-js.github.io/mermaid/#/.

o man I was just looking for a way to not pay for LucidChart. I also refuse to use PowerPoint

draw.io is a solid option as well

You could try OpenOffice/LibreOffice Impress.

"as code" because it's version controlled? Is it also executable in any sense?

No, this just generates diagrams from code. I think it is a pretty neat idea for those who don't like drawo.io or Visio style charts.

The code is really just a pythonized version of the dot format. The real innovation is the fancy-looking icons. When using graphviz directly we have to futz around with images in the filesystem alongside the dot file, in order to avoid simple boxes.

> The code is really just a pythonized version of the dot format.

Not exactly. By utilizing Python as a base language you gain access to control structures as `for` loops, `if`-s and functions. It can make your diagram script much more powerful and/or more complicated.

I think because it's built imperatively?

It’s interesting that PlantUML wasn’t mentioned... there are cloud icons for it as well

The icons are Amazon's own ones. You can use them with any tool.

Reply on news.ycombinator.com