Show HN: Vespene – My new Python CI/CD and automation server written in Django
docs.vespene.io
docs.vespene.io
Does this mean that the CI config cannot be checked into some sort of version control? That was one of my biggest issues with Jenkins. Someone would change something somewhere and suddenly my builds would fail, and even if I track down what setting is causing the problem, I have no idea why it was changed, who changed it, what will happen if I put it back, or even what it's previous value is.
Additionally, it made duplicating a given workflow an extremely tedious, manual process.
Fortunately, we're mostly moving to GitLab CI which doesn't have most of this bullshit.
(Another fan here)
That seems to take care of who changed, what changed, etc.
- It would often get into an unrecoverable state, and we would have to SSH into the box to clean things up to allow it to continue (and in the meantime we would lose all history).
- Trivial automated changes from another plugin would flood the log. Not this plugin's fault, of course, but Jenkins is unusable without lots of plugins, most developed in isolation from others.
Everyone I've talked to about this prefers the GitLab model of telling the server everything relevant to your build at build time. Where Jenkins would need copies of jobs or synchronized configuration updates, GitLab just assumes that each commit has the job configuration it needs.
That all being said, I think you have 95% of that right there today with http://docs.vespene.io/importing.html
This is a JSON file, and it can define what pipeline a project is part of, and what stage it is in that pipeline.
So it's like "this project is part of the analytics pipeline, and it is part of the deployment stage".
But you still have to make the analytics pipeline in the GUI right now. This means that you have to say (and this is all) the analytics pipeline has the following stages in order: A, B, C, and then D.
I think it's quite possible to have the definition OF the analytics pipeline in the vespene.json too, but it's a bit of a question of which one wins if there are conflicting definitions.
If you want to stop by talk.vespene.io this would be a great thing to bounce ideas around on!
All can be built through their UI, then exported to yml in your repos. Pretty good UI. Their service connections are good.
I would implore everyone implementing CI/CD to do the opposite. If you have a small project, maybe use a build script to get it off the ground quickly. If you have a large organization with complex, disparate tech projects, do not use build scripts.
Use CM or orchestration tools. Use pre-rolled containers/images. Use configuration options for existing solutions/plugins/deployment tools. Write only open source, composable, extensions/modules/plugins, with yaml/json/xml configs to control them.
The open source bit is actually crucial. By making all your build/test/deploy plugins open source, you force yourself not to include proprietary information, which is always a dark pattern. Hard-coding IPs/hostnames, credentials, product names, regions, etc just makes your work less composable/reusable. Open source all your pipeline code except for your configs. Then everyone in your business can find everyone else's work simply by going to GitHub and looking up a tool. Because they won't be scripts, they can be picked up and used immediately by any team, and not have to be forked by every team that needs a modified build script.
To do all this properly requires discipline and training and research and time, which big companies have. The point is not to be fast, but to be reliable, to prevent muda from sapping your productivity.
If you want to keep your data externally, there are some good options for this. I'd read the plugin docs about variable plugins, that can easily source variables for things like etcd or Consul.
A script for a given repo could be nothing more than "make go", and that actually is a pretty good practice, to keep that in source control.
Still, you need something to build your code, and to have a good place to see the status.
Ultimately, code still needs to be built, and orgs do want to avoid images they cannot easily recreate. That's the role of a build system and making sure the process to create those artifacts is in source control.
I should also mention that your build script CAN be sourced from source control.
In your .vespene file, just say "script: <path>.
However, the variables in that file can be evaluated by Jinja2 variables, so for instance your security team could set up a snippet everybody could use or your feature flags could be defined from there.
Another cool feature of Vespene is in each project build root all those variables appear as vespene.json files, so it can be a good tool to use to launch all kinds of automation from a web console where you want to record results.
I thought I'd share and if you have any questions, let me know!
Then, what sort of happened is I kept running into overly complex Jenkins installs. The typical miles of config checkboxes was overwhelming for me, but a larger problem was that organizations would have like 200 microservices projects, and all the build scripts would be slightly different.
The idea was then to take the templating system from Cobbler - which allows lots of variables merged in at various levels, and Jinja2, and allow reuse of build scripts through variables and snippets.
There was some other frustration with Jenkins - the idea that it was a ginormous codebase and still not database based, and you had to hunt for good plugins.
Ansible was pretty successful for getting everybody together to contribute to common modules (it maybe took on too many though) and make sure you didn't have to go on too much of a plugin hunt. So could possibly we start a "batteries included" build system?
Finally, I wanted to try out some opportunity to merge the functionality of a build system with something like Rundeck, so you didn't need two tools. I also found Rundeck complicated to set up, but if you want to run some self-service automation, most people do that within their build system. So I figured I could add in some SSH and Q&A interactivity in there to easily get that going.
My other idea was to write a VM/container controller, and while I think there still needs to be a simple one as Kubernetes is getting to be a beast, that's too much work to bite off without a lot of help :)
Honestly the main thing is I just wanted to work with a lot of the same open source folks again (and a bunch of new ones), so I'm really looking forward to that!
I'm curious what you think of https://www.nomadproject.io/
One other reason I didn't build something is I don't have a current need to use something like that, so I'm probably not the one that should be developing it. There's a lot of datacenter use cases I'm ultimately not super familiar with.
(BTW thanks for ansible, it's awesome :) )
I'm used to seeing pages of checkboxes and can't stand how obscure some of the configuration is, but mostly I wanted to write something that could handle configuration differences between hundreds of projects, so that is why there are things like Snippets in Vespene.
Plus, I wanted to make something that was a little better for ops use cases (so people don't also have to go pick up something like Rundeck), so that's why there are things like the SSH integration and http://docs.vespene.io/launch_questions.html
It's more about capabilities and future capabilities than the frustrations IMHO.
Though I do have a bunch of friends who are frustrated with plugin compatibility, plugin hunting (we're doing more of a "batteries included" approach like I ran with ansible - just with much less modules), and stuff like that.
I was also able to add in some stuff like container build isolation really easily, and that's all included stock.
I also saw a lot of people trying to do ops style stuff from tools like Jenkins, which is why Ansible has built-in SSH automation so it can hold on to SSH keys and use them in your behalf, and has some really cool built in RBAC so you can decide who can run what and it can be different than who can edit what.
Mostly I just want to build an architecture that can go interesting places - what is here today I think is usuable, but it's just a starting point. My thinking is if you make an architecture that is really pluggable, and the code is easy to read and add to, it can go to some really neat places.
I'd just recommend taking a look at the various chapters of http://docs.vespene.io/ to get a feel for features, and if you like it, spin up a copy using the setup instructions and see what you think.
Refined framework (oh how I hate that word now thanks to web developers!) went into production in 2009 on Solaris 10, depending on advanced AT&T SVR4 packaging features. It was then subsequently ported to CentOS / RHEL RPM and went into production around 2011. It's been the corner stone of configuration management for the entire infrastructure, particularly production ever since.
While I "grew up" as it were, believing in RPM, we live in a very post-distro, multi-language kind of world, and there's a need for things to glue that together. How many times have I tried to campaign against "wget tarball" as a deployment mechanism, I don't know. It's rough and yes, there's a lack of discipline in ops that needs to improve.
Immutable systems is a VERY interesting way to solve that, but it doesn't work for certain stateful things and you always need something to deploy the undercloud.
I'd encourage you to try to build your own experimental project to try to find different ways to do it, as this is the only real way that technology ever gets ahead.
I have my own configuration management framework which can add and remove configuration from files that each package brings on. It's written in AWK. No knowledge is required to use it - configuration packages deliver just their configuration files' excerpts. In addition to being able to template them with regular shell variables, self-assembly is supported as well.
I wanted another Science Fiction reference to something I loved.
Vespene gas is very important in build order.
My probable worst name was Cobbler (the PXE server) for those that were into cockney, but it was originally named "Shoe", and the idea was a Cobbler makes boots.
Lo siento!
For those who don’t understand it may sound like “(you) see penis”. But to be honest I didn’t even make the connection when I read it the first time.
This looks like a tool that could be a lot closer to what I want to do and designed to do it that way.
Also, is the repo being moved somewhere? https://github.com/vespene/vespene is 404.
Still hard to keep myself from typing that :)
If you would like to get involved with Vespene at a code level, “fork” the project at github.com/vespene/vespene.
So, you might consider a protocol for builders to report their results (e.g. over SSH) without being a full node, if that doesn't exist. Looks good!
Any node typically would run the webserver because it's not that heavy, but doesn't technically have to. It could just run one worker process. Right now the workers do need database access and I suspect that will stay that way for the short term.
Windows needs to happen, but I'm not sure when it happens. (Also a good list discussion).
I'm a bit curious about your raspberry pi use case - maybe a good topic for the forum? http://talk.vespene.io/
I have been interested in building a general purpose CI/CD tool but this part seems like the most "grunt work". Is there an open source library that standardizes webhooks? Or would you be interested in publishing that portion in a library?
[1] https://github.com/vespene-io/vespene/blob/master/vespene/mo...
It seems like you're saying I should have used a many-to-many there rather than 7 FKs. Probably, but shrug... it can be fixed later if it ever becomes a problem.
I hope people know my history and how I've run projects in the past.
But what this does for me is keep me from releasing anything open core. With ansible, ansible was open source but the GUI was not, and we had to hold something back.
The traditional alternatives to OSS are to make a support business, which somewhat encourages making buggy or hard to install software, or a consulting business (IMHO, same) and all of those detract from building a product, collectively, with some of your favorite people on the internet.
So basically Vespene is going to be like a pseudo-foundation.
The structure for this is described here - http://docs.vespene.io/partnership.html - which keeps consulting and support totally free for small shops, and encouraging larger ones that could potentially make millions form Vespene to give back a very small amount.
What normally happens is large companies close six figure contracts supporting open source, and they don't give back a dime, and my goal here is to essentially make a developer's salary off of this project by having those who make more give back a very small amount.
This seems pretty fair to me, but I also recognize that some people don't agree with it.
That all being said, it's not a "no-commercial" clause in any way, and still free for small consultancies, so I'm not anticipating any major problems.
Most of the time, software gets installed in a place of business for that business.
For some of my previous thoughts on this, I'd refer folks to https://medium.com/@michaeldehaan/why-open-source-needs-new-... and later my current thinking https://medium.com/@michaeldehaan/going-with-the-commons-cla...
It's not something I've considered lightly, but this keeps MORE software open, and for a small one-person shop, I think that's more noble than trying to hold some code back and not release it at all.
Some people just want 100% free, gimme gimme, etc - and fine - you're entitled to pick one of those things. That's ok.
I personally view this a little more pragmatically, under the view of fairness, and also have taken steps (in the CLA, etc), to guarantee that trust is never going to be abused.
For instance, the license reverts to pure Apache if Vespene were ever to change hands.
On the downside, I can't get screwed over by IBM or Amazon while I'm trying to crank out free software for everyone. That seems fair too, seeing the time investment running an open source community and working on the code takes.
>I'd argue the definition of what is open source is up for debate
It's not up for debate. It hasn't been up for debate for years. There are two generally accepted definitions of FOSS (each addressing the F and OS respectively):
https://www.gnu.org/philosophy/free-sw.en.html
No one disagrees with these who isn't trying to sow discontent in the open source community. Disagreeing with the OSD or FSD is akin to denying climate change.
No discontent. Quite the opposite - providing some options for less things being open core while trying to earn a small amount on my work.
If people want to call that not meeting some definition, I'm ok with that, but ascribing malice to it seems a little harsh.
No. Everybody who disagrees with these has been driven from your ideological bubble. "Open source" has become a general term that is part of the natural language; nobody has the right to demand everybody use their preferred definition. Many people who use the term will never have visited gnu.org or opensource.org. Many others will have, but will have concluded that the strictness of those definitions is silly. It seems quite reasonable to disagree with a definition of "open source" that makes it practically impossible to make a living off of your work. If you want to have a debate on the merits, fine, but running around telling people they're using English wrong because they disagree with you is just foolish.
You might disagree with the principles of the definition, or think that the definition isn't useful. But that doesn't make it any less of the definition. If you want to do something different, you need to call it something else or you are lying to people.
This argument can be applied to your definition just as well as it can to my definition. Which underlines the pointlessness of arguing over the definition of "open source", and therefore the pointlessness of telling somebody that their thing is "not open source".
Although I'm not sure why I bothered typing that, given the current readings I'm getting on my troll detector.
We don't like open core but proprietary software is the best way to make money, so I'm trying to push that just a tiny amount.
If you are interested, there are two big reasons why my organization would not consider adopting it:
1) We are culturally and organizationally aligned with the open source movement.
2) (More widely applicable and practical) Risk Management. If IBM takes Ansible in a direction that is not aligned with our needs, we can fork it. More importantly- we can pay some one else to fork it, or sponsor a community of developers who want to fork it. Those developers would be under no obligation back to IBM to ask for a Partnership agreement to be able to support the fork.
You stated that "the license reverts to pure Apache if Vespene were ever to change hands". I did not see those terms, and that is definitely something to consider, but even with that, your organization may still make decisions that aren't aligned with a subset of the communities interests.
As an aside, thank you very much for creating Ansible, it is a fine product! Good luck with Vespene, I hope you find great success with it.
At least, I don't think I'm evil. Some people might not know me, but I don't feel evil :)