System Intiative is generally available | Hacker News Reader

System Intiative is generally available

systeminit.com

137 pointsjen2088 comments

The question that immediately comes to mind is:

Suppose that for one reason or another, I want to migrate off of the SI platform. Am I able to get any reusable IAC out in some form? Does SI provide any ways to migrate out of the platform? Or do I just have to rebuild all my infrastructure from scratch outside of SI?

You can export your workspace, and import it into another version of SI. But we aren't producing IaC under the hood - we have a high fidelity modeling layer, and then we allow you to program those models directly.

But if you move off of System Initiative, we don't impact your resources at all. You can just stop using it.

Think of it this way - if you want to "move off of terraform", you're existing IaC isn't useful either (because you need Terraform/OpenTofu to run it). SI is the same.

The main reason I ask is because I use a competitor? of yours currently, and one of the big draws of it is "if we ever go out of business, or if you're dissatisfied with the product we offer, or ..., you can just take all your Terraform and keep using it. You aren't locked in to the platform."

I get it. System Initiative is 100% open source (https://github.com/systeminit/si) - if we go out of business, or you are dissatisfied with the product we offer, you can just take your workspace and keep using it.

We don't make a free distribution of System Initiative - but we expect that someone will eventually, and you could use that.

How do you control/diff System Initiative resource changes with git? I'm not a fan of GUIs for infrastructure stuff since it's usually harder to review, automate, and roll back to a version other than N-1.

You don't, because we built the functionality in to the data model - you can review changes (in multiplayer!) and automate things directly in the application. Usually when we're talking infrastructure code, you don't really roll back to an N-1 version. In SI, you would make the changes you want in a change set, it would tell you if it looks like your change would work, and then you would apply the change set to run the actions needed.

Can you review the changes async or over email?

You can show up whenever you want to! Today it happens in the UI, but we could certainly send you a diff in your email at some point. :)

How do you keep an annotated history of your changes? Using revision control is not about rolling back, it is about knowing why things are the way they are.

It's a good question! Having a history of how things transformed over time matters. We keep track of that in a couple of ways - one is the graph snapshots themselves, and the other is a raw history log. It's really just another view of a snapshot of the graph at a particular time, and a delta between that graph and the current.

We have lots of planned work coming here - but we have a very rich dataset to do it from, and we're stoked to get there.

Having the history of changes is a step, but the important part of that is the why those changes were made. Git can capture this with good commit messages. Do you have any plans to capture that "why"?

Yep! The application of Change Sets is a similar moment to record the "why", alongside all the information about what was done. We have more to do here, along with adding things like RBAC that requires review before change sets can be applied based on ownership of resources, etc. But this is the plan!

Thanks for answering my questions! Look forward to playing with it.

Another nice thing about using code is you can easily include comments explaining why something is done a certain way, right next to the code that does it and thay is also included in the history.

This is a very bold take on Infra management which has become a real PITA even at small orgs - Cloud really did a number on us.

One thing I'm personally wondering about is whether I can import my Terraform state file - because that'd be a pretty good starting point for many orgs.

Regardless, I'm curious how this pans out. Though we've had a few different iterations of IaC in the past decade or so, the infra crowd has been known for being sceptical when it comes to adopting new things than your usual software engineer, especially something that is more like a step change than a gradual evolution.

Very happy someone's taking on this task with a very fresh approach.

You can't import terraform state right now. High on the road map (https://docs.systeminit.com/roadmap) is discovery of existing resources, starting with the import of individual things, and eventually building the component backwards from discovering the resources. We think that'll be better than importing terraform state, because it will essentially be a rolling reflection of the output, rather than trying to back into it from terraform.

But we're 100% open to importing it from the state file if that's what folks need.

I would be fine with discovering the existing resources actually, if SI can do that. The reason I mentioned Terraform is because my infrastructure is fully managed by it so I thought that would be the easiest path, but if SI can automagically discover all the resources that’d work just fune for us. Thanks, need to give it (SI) a go!

This worked in a previous version of SI (we've built like 4 different versions of it over the years) - but we haven't brough it back in to the current version yet. But we absolutely will!

Importing from the state would be good.

Doing this with resources is going to be the easy path. Translating the state representation of re-usable modules I expect is going to be more difficult but necessary for the migration path to be useful. A lot of power in Terraform is being able to stack re-usable components, say an internal module representing a service on top of a public module that provides sensible defaults like the terraform-aws-modules collection which then sits on top of resources.

If you can get this to a place where an existing deployment of Terraform modules can be imported into SI as a collection of re-usable templates and then users can easily deploy n+1 of those templates the same way they would do with Terraform I think you will have an easy migration path.

If not importing from the state at least some way of automatically importing those re-usable components and allowing the deployment of n+1 of a reusable component would be helpful to migration from existing patterns.

I think you should also talk about the drawbacks, potential roadblocks and failure points. Otherwise it's not possible to make an educated decision on whether to move to SI from, say, Terraform, and as such, we wont. SI is very early days and it's exceedingly obvious there will be a ton of issues with it. Terraform has a ton of drawbacks and it's old tech by now. Provide us information so we can decide whether it's worth dealing with SI issues to no longer have to deal with Terraform issues. Otherwise this is just marketing speak and will fall on flat ears.

All marketing is marketing that will land on flat ears. In the end, you'll have to try System Initiative, see if it is a fit for your use case today, and if it isn't, if it's worth paying attention to tomorrow. I wouldn't (and you shouldn't either) make a technology decision based on what anyone says on their website or blog. :)

Today the obvious drawbacks:

* Terraform has tons of coverage in their provider ecosystem, and we're not close to that yet.

* We have some enterprise features still to add.

* There is some work to be done around huge infrastructures, both in how to provide easy ways to visualize them and how we scale the underlying graphs.

https://docs.systeminit.com/roadmap/

We have plans for all these things, but it's early days. My advice (not just for SI) - you should always build representative prototypes if you want to understand what a technology might do for you. Your circumstances matter, and your problems are likely unique.

Prototypes are expensive and as such difficult to justify if the technology doesn't look promising. I'm sure you're aware there's a new self-proclaimed miracle tool appearing in this ecosystem every day. My point is that there is a severe lack of information to make an educated decision here.

I think it's fair to say most people will be interested in potentially replacing Terraform with this. Do you have a comparison against Terraform? Is there a guide on how to import resources into SI?

Rest assured, we'll have more competitive content over time. :)

Having a pros/cons page vis-a-vis alternatives that was honest and thorough is something Hashicorp always did well, to their credit. You should, too. It earns trust.

One of my past employers specifically declined to write up this while trying to sell into a crowded space like SI aims to. In my case I really got the stinkeye for asking about it (how we compare, and do we document that publicly anywhere) during my first few weeks. It definitely left a sour taste for me.

Thanks for the feedback! I appreciate it.

From the FAQ:

> When you turn infrastructure into data rather than code, you obviously don't want to stick it in Git

Why not? I still do.

Please add mote context to your answer, otherwise this is just another way to say "why not?".

This exactly my point. The FAQ makes the assumption that people don't want this, with no justification. I'm interested in what that justification is, especially given that the state-of-the-art that this is competing against is Infrastructure as Code, where storing this stuff in git is considered a good thing.

Does it have composable, reusable, parameterizable components that allow you to set up similar infrastructure in multiple environments or regions?

Does it have a way to add support for underlying infrastructure that isn't natively supported, similar to terraform provider plugins?

How do you plan on keeping these "simulations" up to date, and consistent with real infrastructure, especially as you add support for more cloud providers?

To your first question, adding what we call management components is high on the to-do list. They serve this role, in addition to letting you describe workflow across the other infrastructure they manage.

You add support directly in System Initiative, by writing the schema and functions in TypeScript. You can see how in the docs.

The resources can be periodically refeshed, and the data flows through all your open change sets automatically - kind of like an automatic rebase. This is also an area where we’re doing work - both to figure out the right intervals and to show drift more clearly.

It is exciting to see progress in this space and new ways of thinking infrastructure, kudos to everyone!!

My only hope is that we learned and we don't end up managing things like GH repositories or Pagerduty schedules.

Hello! Adam from System Initiative here. Happy to answer any questions. :)

maybe it's because i'm not in the devops space but i'm 4 paragraphs in and i still don't know what this is about?

Perhaps https://systeminit.com would make more sense. :)

Not to me. It says "System Initiative is an Intuitive, Powerful, and Collaborative replacement for Infrastructure as Code" but I still don't understand what it does or why I would want to use it.

They get to it if you scroll! I came to the comments with the exact same concern, but they eventually spell out the value-add in (IMO) very clear terms:

  First, we turned everything into data - a rich system of digital twins that enable safe, easy simluation of changes, and map 1:1 to the upstream resource.

  Second, we made it all programmable and reactive by modeling it on top of a hypergraph of functions. When one property on a model changes, anything dependent on that property automatically re-calculates.

  Third, we built a multiplayer user interface that makes working with the model fast, safe, and fun.

Seems cool, though I'm far from needing it. AFAIU: it's an attempt to build a reactive infrastructure system from the ground up using a cute Functional Programming paradigm, not just building off of code-like YAML files like we do today.

Personally the GUI gives me flashbacks to the dark days where "programming" in my mind was "using Eclipse", but that's a biased take! I can see why WYSIWYG-style functionality was considered too fundamental to make optional.

The trick is that we’re not replacing programming with the UI - we’re just making it easier to compose things together. Plenty of code in SI - just not the way you write IaC today.

Can that code be versioned in git?

Or is it just a blob in the database for SI?

I read through the blog post and entry page twice each - yeah still confused. The pitch reads like an engineer celebrating their implementation.

What I took away is: it’s a collaborative IDE for infrastructure? with some nifty simulators to catch issues earlier, and “somehow” changes are managed outside the popular git+pipelines workflows?

There are elements of this that I like (faster validations that CDK deployments ). Those aspects are bundled with confusing, either unnecessary or poorly communicated, other elements. “Replacement for IaC” - is there a new paradigm? Or is IaC just now a graph in this local application? Because you tout being able to program new service models, so the code isn’t gone…

You're not alone. I am in the devops space, and about half way through I was thinking "ok, I get you're excited about this thing, but what is it?"

It looks like it is in the same space as terraform, but uses a gui instead of defining your infrastructure as code, but allows you to "simulate" and review changes before actually applying them.

First, I have to say this looks awesome. I am in awe at the level of effort I imagine is required to build and maintain something like this.

That being said, I find the rationale a little bit confusing. I rather love IaC, and consider a GUI or no-code/low-code tool to be more of a dead end (not for any fundamental reason, but for more practical reasons) than plain text. I do really appreciate the problems solved by the simulation approach, but to me these two things are orthogonal. I feel like you could have a product that functionally does what your product does, but with a plain text interface. I appreciate that you're really going for something different here, but I am sure I am not the only person who feels this way.

I don't think you're alone, and to be honest, I'm as suprised as anyone that it turned out to be better. We built a lot of different implementations on the way here, and all of the initial versions started with code and plain text as the interface. But it's very hard (I think impossible) to change the user experience when you do that, because the data gets locked up in code - there isn't a good way to "see" what the real world is like, or what your proposed changes would do.

But you're not alone in thinking this, and I completely understand why you would. The history of things that look like this in this space is.. not great. :)

> the data gets locked up in code

I'm really not sure that this is a bad thing.

> there isn't a good way to "see" what the real world is like

If the IaC system had the same under-the-hood functionality as System Initiative, what's to stop someone from also building a GUI visualization of the IaC-code?

Think about it this way - how do you update the IaC from multiple places? How do you visualize drift? IaC grows like code grows - it gets factored differently, it gets abstracted, etc. It's easy enough to imagine how you would update a single declaration, but thinking about how to make the "code" reactive will break your brain.

So what we do instead is have a reactive data model, and shift the code part to a reactive graph of functions.

> How do you update the IaC from multiple places?

The same way you update any other code from multiple places. A version control system (eg.: git and github), with CI/CD.

> How do you visualize drift?

Why do you have drift in the first place? Gitops is an obvious solution to drift.

> It’s easy enough to imagine how you would update a single declaration, but thinking about how to make the code reactive will break your brain.

I’m sorry, I don’t follow at all here. I’m not sure what the problem is with IaC. If your IaC is declarative, it’s no more complicated than data.

I'm not a Tik-Tok child and have an attention span long enough to focus on coding longer pieces. However, I shouldn't be required to read all this wall of text to understand what this product is about. Could you summarize it in 4 sentences: what it is and why I should use it (instead of X or Y)?

How about 1 sentence? "System Initiative is an Intuitive, Powerful, and Collaborative replacement for Infrastructure as Code" from https://systeminit.com

That sentence reads to me as "the product has been described using marketing."

(I don't have a better one sentence tagline as such, mind, but honestly if I'd only read that I'd never have bothered to look - see my sibling comment for an attempt at an actual high level description ... especially since I probably got something wrong you'll need to correct ;)

Your comment is GREAT. We were all really impressed with your interpretation, thank you very much!

Amusingly (to me at least) that was based just on the blog post and zooming in to the screenshot at the top.

However, thank -you- very much for the response and you're absolutely welcome.

Please consider the comment to be licensed under the union of all OSI approved licenses [1] if you want to steal and/or improve any of the wording.

[1] The debian ftpmasters once complained about my having released something to CPAN with two licenses inside. I asked if they wanted the next release to be explicitly under said union so they'd have to tag the upload with all of them. They decided their complaint wasn't actually that important after all.

(and if that makes you think I'm a monster ... ask Adam to explain just how right you are about that ;)

"Imagine if you took IaC, bolted a slick Native Objects style frontend onto it, made it collaboratively editable and integrated version control and - perhaps most importantly[1] - a high fidelity preview simulator so you could look at the total expected results of your configuration changes without having to wait for AWS/etc. to finish catching up."

[1] To me at least.

So, my hot take on this is that it's an advanced GUI for AWS with change management built in.

1. Is that a good summary? 2. Why would I pick this?

I know you LOVE it, it's your baby. But why should I love it? :-)

It's not a bad summary, but it is a shallow one. :)

You should love it because it's a more intuitive and more powerful way to build this kind of infrastructure automation. What's happening under the hood isn't just infrastructure as code with a UI - it's a full reactive model of how things work. That's what makes the UI possible, but it's also what brings about so much power - the code that drives those models is also fully exposed and versioned.

So when you have something like a policy to write, you think about what resources you need, use them as inputs to the function, and then store the results. Check out what an early user had to say about it: https://matthewsanabria.dev/posts/take-the-system-initiative...

We'll find out if you love it or not. :)

I can see why you went full GUI to begin with now.

The example component code reminds me a lot of mobx-state-tree (you have no idea how much cog. diss. I get reading the docs for that thing given they acronym the name everywhere ;) though I find myself much preferring the API shape of mobx-keystone at this point.

(I've been experimenting a lot with reactive graphs of late though while it seemed an obvious thing to try at some point I haven't attempted to wire it up to systems automation yet; shall have to do my usual cover-to-cover documentation read on your site and then hopefully I'll be in touch with a baseline to actually chat about that part ;)

I mean, we didn't start there - we wound up there :)

But yeah, you're not wrong that it's got a lot of inspiration from things like mobx and rxjs.

Yeah, I guess I meant "for release 0.999_001" [1] or something rather than a strict interpretation of "to begin with."

> But yeah, you're not wrong that it's got a lot of inspiration from things like mobx and rxjs.

Please figure out how to deploy a subset of the reactive graph into a k8s operator. It'll be a really cool feature and also it'll probably save me a bunch of time when I want one of those if I can crib from your work :D

I ... bah. I am really looking forwards to getting into another of our involved coversations about this stuff but I'm too tired today and besides I definitely do need to mainline your docs first. I'll probably see you on twitter first with the assumption you'll end up chasing me onto Discord sooner or later ;)

[1] please interpret that in terms of how you remember me (ab)using version numbers ;)

Not to be too negative, but:

> When modeling AWS IAM policy in System Initiative, we realized that AWS provides a sophisticated Policy Simulator. So we modeled it, connected our IAM Policies and resources to it, and had a new, real time interface to test the validity of IAM policy. It took less than an hour from start to finish.

Clicking the link takes you to the docs on policy simulator, which seems to show it’s quite limited and isn’t representative of actual, deployed IAM rules:

> Important:

> The policy simulator results can differ from your live AWS environment. We recommend that you check your policies against your live AWS environment after testing using the policy simulator to confirm that you have the desired results.

https://docs.aws.amazon.com/IAM/latest/UserGuide/access_poli...

It's actually pretty good - usually the reason it's not accurate is because enough data isn't being fed to the simulator. That's one of the things that was great about doing it in SI - it wasn't hard to get the data in to the simulator.

But if I was AWS, I would also say you should check your IAM against the real world, because if you don't, it's pretty easy to wreck you environment. ;)

For other resources that don't have a simulator provided by AWS (e.g. an EC2 instance) how do you ensure that your simulation is accurate? How do you keep the simulations up to date?

You write a function. As an example, here's one way to check if you have a valid AWS credential

``` async function main(component: Input): Promise < Output > { const authCheck = await siExec.waitUntilEnd("aws", [ "sts", "get-caller-identity", "--region", "us-east-1" ]);

    if (authCheck.exitCode === 0) {
        return {
            result: "success",
            message: 'Credentials are Valid'
        };
    }

    return {
        result: "failure",
        message: 'Credentials are invalid. Please check the credentials set on the secret/credentials prop!'
    };

} ```

Checking the validity of an STS token requires that you have a bona fide token first. Thus it’s not a “digital twin”; it is the real McCoy. And STS tokens are free.

There is no digital twin I’m aware of that is capable of simulating the real behavior of an EC2 instance. There are just too many variables to consider. To test instance launch and runtime behavior to a meaningful degree of certainty, you have to launch one first. And that means accepting the costs of doing that.

(I notice, too, that you appear to be executing the AWS CLI to do this. I’m not sure if that’s bad or not, but it smells a little fishy.)

It’s terribly slow, given that it’s starting an entire Python process, configuring boto3, etc. that’s 2 seconds on my machine, just to run —help. And it’s all to make a single HTTP request (80ms)

Not sure if this is just pseudocode though.

You can use fetch if you want to call the API directly. It is just an example.

Yeah I figured! I like the design, it’s pretty cool.

We're being intentionally pragmatic here. If you're building a digital twin of, say, an F1 car - the complexity of the simulator has to be very high. It's more like building a mock of physics than just the car.

With Infrasturcture, it turns out that what you need to know is "did I make a valid configuration", or "does this set of things work together". It's less about making a mock of the results, and more about simulating that the results would have the effect you think they will. So we can't tell you "will your application work on this size of instance" (although if you know that, you could encode that!) - but we can tell you if the options your setting are correct, if the AMI exists in the region, etc etc.

If you need any help, you can reach out to me at pravanjan@palette69.design

1. Would you still try to pitch this to someone who is violently allergic to "clickOps" and GUI-centric paradigms? What would you lure them in with?

2. How is a "multiplayer" experience superior to code review-based collaboration flows, in general and for infrastructure work in particular? (I'm someone who values peer review highly and thinks that an independent reviewer improves both the credibility of the process and the quality of the outcome, but who doesn't think a pairing session counts as an objective review.)

1. I'm sympathetic to your doubt - I had the exact same doubts, and part of why it's taken us so long to build is we refused to sacrifice what was good about IaC to get "simplicity". If this was ClickOps, they should be allergic to it. But it isn't. Under the hood is a very powerful new primitive - a reactive hypergraph of functions. The UI is there because it's a fast way to compose things together, and to radiate more information than you can get from an editor alone. But everything you do is tracked, it's fully open and extensible by writing code, and it's much easier to use to communicate with your team members who might not know all the intricacies of IaC.

2. There is how it is better today, and there is how it can be better in the future. Focusing on just today - frequently the kind of review that needs to be done is by an external subject matter expert. Being able to bring those people in to a change set, show them what the change you are proposing is, and have them inspect and alter it with you in real time is great. An example here is one of our early users wanted to use ECS, but had never used the service before. So they put things together in SI, asked someone who had that expertise to look it over - they could see the architecture, they could change properties, add a few missing things. It was much more straightforward than a back and forth in a PR.

But that's not to say that, in the future, there isn't more to do. We need to have more functionality around who needs to review things, build more specialized views for the review (there's no reason you should be stuck doing a review only in a single view of the architecture), use the snapshots we have of the entire graph to build more insightful ways of communicating what's changed (and what actions will happen when you apply.)

Think multiplayer and powerful review and approval semantics.

If I have existing infrastructure, are you able to generate a diagram/model of the current state given sufficient permissions?

This is a great question. Eventually, yes. In an earlier prototype of SI, we actually had this feature, and it was pretty dope. We removed it as we made things much more programmable, but it's high on the road map to bring back. The first will be an `import` function that just builds an individual component from a resource, followed by the full discovery feature.

Naturally, what -I- really want is a three way diff between "the state SI thought production was in, the state SI has been asked to transition production to, and the state production is actually currently in."

I recall trying to convince you to experiment with that when you were building Chef but you'd just come out of working with finance stuff so understandably felt that an uncontrolled change should always be dealt with via emitting a resume generating event.

I continue to believe that for small non-bank organisations, when somebody gets paged in the middle of the night "whatever gets production to stop being on fire the fastest" is completely legitimate and systems automation tooling should support handling the config reconciliation -after- it's back up.

... but enjoy your launch day, having waited this long to argue my case again I can leave it a while longer :D

This seems very interesting, but the resource-hour pricing is a total nonstarter. You're incentivizing me to use as little of your product as possible. You want me to pay for every bucket file, every role assignment, every verification record?

Charge per user or a percentage of cloud spend under management.

DOA at my current employer because of a lack of support for other clouds (GCP and Azure). I'm sure improved API support is in the pipeline though, very psyched to see SI grow!

It absolutely is. We're adding more resources all the time, we hang out in Discord and build the things folks need most. We're working on GCP now.

Love the cookie settings section that provides a lot of detail about what cookies are used under what category, and what they specifically do. Never seen that before but it immediately raises trust with the website/organization. I hope this catches on more.

(Off-topic to the post, but since the discussion seems to be dying down anyway, this seemed worth commenting before the post disappears to the back rooms of HN.)

I love seeing innovation in this sector. I’ve long felt that there must be a better way than terraform and pulumi. Excited to try it out.

Thanks! Let us know if you need anything.

Looking at the website, this is fully open source. Nice! Looking forward to trying it out.

Yeah, I'm not gonna click around in a GUI to provision infrastructure. Not groping around a console was the whole point of Cloudformation and Terraform. While the UI looks cool and impressive it's a massive step backwards. I can't codegen and document a GUI. Based on the Tutorials it doesn't really solve the 200% problem because very little of this seems intuitive. Which is fine, it's a tool for professionals, investment is an expected exchange for power but I don't think it reaches the bar of "don't have to learn another thing."

The simulator, if it works as described, is huge. Would be worth 5 figures/year/engineer in time saved alone. If I can deploy to a simulator and it's close enough to the real thing that passing means g2g to prod the you have a gold mine. Oh lord, if you could write tests against the simulator and bring first-class unit testing to devops I might cry.

Excited to launch this! All of our early users feedback has been consistent: going back to Terraform after using System Initiative would be terrible.

We're here if you have any questions.

I'm privileged to have had the opportunity to help the team build the Production SI SaaS platform, leveraging SI itself to create all the infrastructure. I'd be happy to chat with anyone interested in how it works and share findings.

Maybe it could be called System-I.

I think all great companies have names that are also great band names. System Initiative is a better band name than System-I. :)