Personally I'd much rather see a GUI that can help string together snippets of Terraform/CFN and Ansible/Chef/Puppet/Salt/whathaveyou. A visual IDE for infrastructure automation, if you will.
And it worked both from the command line and the GUI. I can see no issues with it having a web interface as well.
I would definitely find that useful for Terraform and Cloudformation. Searching for it, I see that AWS has a tool targeted at that: Cloudformation Designer.
https://aws.amazon.com/blogs/aws/new-aws-cloudformation-desi...
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
https://userify.com/docs/enterprise/cloudformation/
If you have an AWS account, you can actually modify/fork the template on the fly by clicking on one of the objects in the diagram and editing its properties (which edits the YAML CloudFormation template in the background).
For example: you have Puppet, Chef, Ansible, Salt, etc. They can all make a directory. They do exactly the same thing on almost all systems they support, because POSIX.
They literally all just run this syscall: int mkdir(const char pathname, mode_t mode);.
But each tool has their own goofy way of being told to make the directory. Why can't I just pass "{'mkdir': [ '/the/path/name', '0700' ]}" to all of them? This is literally all they will be doing, and all they need to know. Just make this path with this permission. This is declarative programming, without a custom language!
Anything else, like directory ownership, should be an extra object, like "{'chown': [ '/the/path/name', '1000', '1000' ] }". Again, this is literally called the same way everywhere: int chown(const char pathname, uid_t owner, gid_t group);.
You can combine the two into one JSON blob in the output of this config management tool (that's what this GUI really is - a GUI CM with a dashboard). You can have sets of blobs with tags that are discrete reusable pieces of configuration. And every tool can just read them, follow a spec based on POSIX, and execute them.
- What happens when "/the/path" changes depending on the distribution? How do you specify that?
- What should happen if the "/the/path/name" already exists? Should the system touch it or something?
- What if it exists but is not a directory? Should it delete whatever exists and create a directory instead?
There is no way to define that stuff using POSIX semantics, because POSIX is inherently imperative. "mkdir" means "make a directory", not "on this path there must be a directory".
That mismatch in semantics is real, and each configuration management system tries to close that gap in the simplest, most user-friendly way they can figure out. Most have flocked towards defining their own, as declarative as possible, API. For instance, ansible defines this as:
- name: Ensure name's base folder exists
file:
path: /the/path/name
state: directory
recurse: no
owner: pid/name
group: gid/name
The example above is two basic operations: mkdir and chown. On top of those operations are policies, rules which need to be applied. And then there are transforms, where the arguments or policies may change depending on some variables.
These are all different things that the above rule is mixing together, and we're supposed to think this is a good thing. We pretend it's less complex, but it's not, it's just obscured. And the end result is a lot of wasted effort to redo the same thing.
The GUI is useful because it can craft the operations, policies, and transforms for you, and output instructions to be interpreted and executed by an agent. You could keep all the complexity, but make it standard among a set of common tools. They really all do the same things - they just don't speak the same language.
Another way to think about this is from an app's perspective. It just wants to write to a log file in a directory, it does not care where. That's somebody else's job to figure out - just tell me where the file is! Wouldn't it be handy if the app had a language to tell a CM tool what it needed? It wouldn't have to say "make me file X in Y dir", it would say "give me a writeable file that I will call 'log_fd'".
Then we wouldn't need to craft an environment for it like parents assembling a nursery. The apps could be more self-sufficient, and the CM tools would just do their bidding. Using a standard language.
If the app just asks for stuff, where would the management part come it? And why have a CM at all? Granting resources is what the OS itself does; the application can just call tmpfile().
The CF could modify its behavior to fit a given platform. If you're on Windows, it will do operation X in one way, and on Linux a different way. You could describe to the CF the information it needs to fulfill requests - where are the big data volumes, where are the credentials, how should it connect remote resources, etc. When the app asks for something, the CF will have just enough information to figure out how to provide it for that case.
The app can label resources however it wants. For example, if it called tmpfile("my_logfile_access"), a CF could return it a temp file, and record in some datastore that application X asked for a logfile 'my_logfile_access' and that it was provided. If you want to process logs, ask the CF where the 'my_logfile_access' file/directory is. If you abstract away the need to tag it and just have a function called logfiles() or something, you wouldn't even need to know this tag - your app could just write logs with some tag metadata, so when you go to process the logs, you figure out what it is after you've opened the logs.
Same for data, and ports, etc. Why should I have a Vagrantfile that maps ports? The app knows what ports it wants to offer. And why use numerical ports? We've had /etc/services for like 30 years and yet nobody seems to realize you can use a name instead of a port number and stop worrying about manually mapping numbers around. Honestly, so much of modern devops stuff is backwards.
Some applications need entire configuration languages because they are so flexible that writing some ini/json/xml quickly becomes too cumbersome. How would your system deal with those problems?
Finally, your proposed solution seems to suffer from having to adapt all applications to be able to manage them. This is impractical to say the least, and if you had the resources to take that adventure... I think you'd be better off turning to something like Nix
#!/usr/bin/perl -w
require App_Manager;
app_provides("service", "nginx", "config", "test.com", "server { root $APP_HTML_DIR; index index.html index.htm index.nginx-debian.html; server_name test.com www.test.com; location / {try_files $uri $uri/ =404;}}"
app_provides("service", "nginx", "config", "example.com", "server { root $APP_HTML_DIR; index index.html index.htm index.nginx-debian.html; server_name example.com www.example.com; location / {try_files $uri $uri/ =404;}}"
app_wants("service", "inet", "http", "{ tcp port http ; tcp6 port http }", "acceptable_providers { nginx; }")
app_wants("service", "inet", "http", "nginx")
Ideally the above would not include an Nginx-specific configuration, but one that was more abstract, to allow any HTTP service to provide for the needs of the application. That's probably wishful thinking. But on the other hand, there are probably millions of examples of an application needing an HTTP server in extremely similar ways, so maybe service-specific configuration specifications aren't totally insane.
> Some applications need entire configuration languages because they are so flexible that writing some ini/json/xml quickly becomes too cumbersome. How would your system deal with those problems?
You can abstract away almost all of the details of an automated system. They're just not built today with this in mind. Systems today are built as isolated islands of complexity and depend on humans to teach them how to shake hands. This is largely not necessary, but it is with how people are currently managing systems.
> Finally, your proposed solution seems to suffer from having to adapt all applications to be able to manage them
In the same way that networked applications "suffer" from adapting a standard protocol.
Another example:
#!/usr/bin/perl
use DBI;
$dsn = "dbi:SQLite:dbname=$dbfile";
$dbh = DBI->connect($dsn, $user, $password);
$sql = 'CREATE TABLE people (id INTEGER PRIMARY KEY, fname VARCHAR(100), lname VARCHAR(100), email VARCHAR(100) UNIQUE NOT NULL, password VARCHAR(20))';
$dbh->do($sql);
When I apt-get install most applications, I don't need to tell them where they should put their logs, they just the mkdir('/var/log/...') API. I don't see us gaining much by adding another man-in-the-middle to translate a logfiles(X) call to mkdir('/var/log/app')/open('/var/log/app/'+'/var/log/'+X).
The reason I use a CM is mostly to configure the system itself (such as - which apps I want installed in the first place) and to define application-specific configurations, which a configuration fulfiller would have no way of knowing how to do.
The app knows what ports it wants to offer.
No, the app knows what services it wants to offer; ports are shared resources, so they can't be defined by each app.
We've had /etc/services for like 30 years and yet nobody seems to realize you can use a name instead of a port number and stop worrying about manually mapping numbers around.
On the contrary; /etc/services is part of the old paradigm, in which each application got a static port that you had to manually configure, even if you used a text file to give it a label. In my systems, the CM takes care of that, by allocating any free port and taking of the bookkeeping to prevent double use.
Part of the problem is systems having their own standard. What I'm suggesting would be a standard way of an application defining its own standard, which other applications or systems could interrogate and learn dynamically. It would effectively end a majority of platform-dependence, other than the platform-specific interfaces we unfortunately have to live with.
> No, the app knows what services it wants to offer; ports are shared resources, so they can't be defined by each app.
> On the contrary; /etc/services is part of the old paradigm, in which each application got a static port that you had to manually configure, even if you used a text file to give it a label
You are missing my point. The point is to get the application to dynamically pass a service which it needs or wants to share.
Ports are shared resources, but that does not mean they can't change. They are largely disposable, arbitrary numbers assigned to higher level concepts. We assign them statically because you can't easily bind one port to more than one application, and due to the inherent limitations of most protocols.
There is no reason I should need to tell Docker to "map internal port 80 to external port 18080". I should only have to tell it "I'm running http, please tell whoever connects to you (Docker) that I'm running http, and give them a port to access my http service on".
How so?
> The CF could modify its behavior to fit a given platform. If you're on Windows, it will do operation X in one way, and on Linux a different way.
This is exactly what most CM tools do. Chef is my specialty and in Chef, anything that makes sense to do on multiple platforms will, if done correctly, present a unified interface. Resource providers and community cookbooks abstract the OS-specific stuff from you, the CM administrator, so you can just include high level concepts. When done well, it's as simple as describing the end configuration for an app (generally and for the specific instance you're spinning up) and off it goes.
> If you abstract away the need to tag it and just have a function called logfiles() or something, you wouldn't even need to know this tag - your app could just write logs with some tag metadata, so when you go to process the logs, you figure out what it is after you've opened the logs.
If you look inside of any decently written application, you'll find that they use a centralized logging facility of some kind, which then demarcates the border between application and the running system. The app will provide a means of configuring its logging output to some extent, and then you're responsible for where it goes on the other side. Logs on disk, Event Viewer on windows, journald, or being sucked up into a logging pipeline like ELK, it's really up to the particular use case. Note that already, we're dealing with some pretty seriously different outputs; obviously this needs to be configured somewhere!
> Same for data, and ports, etc. Why should I have a Vagrantfile that maps ports? The app knows what ports it wants to offer.
Maybe the app does, but how should Vagrant / Docker / etc. know about this? The app could supply a Vagrantfile and a Dockerfile, but how does the app know what port I want open in reality into its own little port space? I may be doing something weird and not want the defaults for any number of reasons.
> We've had /etc/services for like 30 years and yet nobody seems to realize you can use a name instead of a port number and stop worrying about manually mapping numbers around.
/etc/services was almost never something you could rely on outside of the major protocols and if anything I'd prefer to just see it stop defining anything above 1024. It's kind of silly to think that there would only be ~64k different services in the world, now, but I suppose in the 70s it was more excusable.
> Honestly, so much of modern devops stuff is backwards.
There's backwardness, but I think some of it is in the way you're thinking about it. I don't mean to be insulting, but a really well managed CM environment absolutely does provide the majority of what you're asking for. It's not all out-of-the-box, maybe it could be, but that's sort of a pie in the sky way of thinking - "gee, what if all the app developers actually coordinated on something!" - yeah, might could work if everyone agreed to get over their own vision/ego and implement a common thing, but that's both unrealistic and also potentially stifling of innovation.
Devops is the job of surfing on top of the chaos and bringing as much order to it as you can so that when you inevitably burn out and leave the company, most of the good work you did has more momentum and inertia than it used to before we implemented common mechanisms of doing automation. Another Chef or Ansible devops guy can slot into your place, figure out the theme, and keep it going - even if it's not how they would have architected it from scratch. There's a hell of a lot of value in that.
For most people, the point of config management tooling is to turn all their manual, bus-factor-of-one crufty legacy processes into automated, documented, centralized crufty legacy processes.
Ops people don't want a system that is opinionated or that pushes back to force them to change their process. They want a Turing-complete system, and maybe macros as well, so that they can quickly encode their hairy manual process into a hairy automated process, and get back to all the other firefighting they have to do as ops people.
I think about cars and planes, and wonder what life would be like if we needed a team of engineers sitting on the wing of a plane to keep it in the sky when the pilot moves the flaps.
It's the world we'd be in if new planes were in such high demand that the number of startups building them totally outstripped the supply of aeronautical engineers, to the point that most new designs only had ~5 man-years of effort invested in them on release, rather than the thousands they get today.
Software is really a different world, economically.
To be really useful, it should have some kind of snippet options (insert a vpc with all the usual subnets, routing, etc in one go), or some "fill in the blanks" functionality.
If you have specific ideas on how would you like a functionality like that to be, I'm all ears to make it real!
[1] https://github.com/serverboards/serverboards/ [2] https://serverboards.io/developers/18.01/plugins/screen-defi...
But I think I see what you mean and I like the idea. A low level GUI that lets you look up specific tasks in ansible and have it create the playbook for you at the end.
That would be pretty cool and not that hard to write actually.
Can't say anything about Chef or Salt, but with Ansible you get basically shit for help since the IDE just sees a YAML file and nothing more. This is partially the fault of Ansible though, to be honest, modules being standalone scripts that is cool - but they provide no standardized way to inspect their parameters and outputs beyond documentation.
See https://github.com/ansible/ansible/blob/devel/lib/ansible/mo... for example:
DOCUMENTATION is a yaml snippet that follows their documentation schema to tell you about all the module's input parameters.
EXAMPLES gives you some usage examples.
RETURN documents the outputs.
A proper IDE plugin could parse those snippets for each module and do some pretty nice autocompletion/inline documentation for you...