Introducing Boxen
github.com
github.com
* Thoughtbot's laptop script - http://robots.thoughtbot.com/post/8700977975/2011-rubyists-g...
* Lunar Logic's Lunar Station - https://github.com/LunarLogicPolska/lunar-station
* Pivotal Labs's Pivotal Workstation - https://github.com/pivotal/pivotal_workstation
I personally liked Pivotal Workstation the best, as it had the best combination of robustness, pre-built recipes, and easy configurability. I'll be excited to take a look at Boxen the next time we bring someone into our team.
That's why I love a product like Action.io (http://action.io), which allows all dev-boxes to be kept in sync and create clones when needed
(disclaimer: I work with action.io team and it's still in private beta)
I guess how useful it is for a sysadmin would depend on whether changes are pushed automatically to the macs.
Github is large enough that they should have in-house IT looking after the desktops/internal network. They might even have two teams, one looking after employees, and one looking after the server infrastructure.
Start-ups grow organically, so there might not be "someone" at first, other than the developer. This quickly turns into a burden when you have 50+ employees running around and warranties start to lapse, there is no current inventory list, you have a mish-mash of machine configs. Then comes the stage of getting a consistent config on each machine, so that person X and person Y can collaborate, using the same dev tools, at the same rev level. Looks like Puppet/Boxen is a tool to solve that problem.
This problem is compounded when you are in a hiring frenzy, because new people almost appear out of thin air, and you are supposed to have a machine for them. Having a quick deploy script will save your hair line considerably.
I am so glad to see this as an alternative to the tools that Apple has been neglecting since they stopped trying to sell server hardware.
On mobile there are a variety of MDM services so you can almost do this for phones, but there isn't much for the SMB market for desktops/laptops. In the Microsoft world there are some tools, but even those are generally too much work for a small business to set up (even a non-tech business with 100 employees isn't likely to do it, at best they'll have ghost or something to image new machines -- a tech business might after 25-50 employees)
Rather than maintaining all this enterprise stuff to fight off the Windows tar-pit of viruses and dreaming of a day of MS Office on Linux, why don't you switch to Macs which require no such special effort to stay virus-free and already have MS Office available?
Am I missing something here?
For a 5-25 person SMB, there's not a full-time IT guy who could set it up. At best, there's a desktop support/printer/office manager type. At a tech company, you might get lucky (or unlucky) and one of your devs or site-sysadmins spends much of his time setting stuff like this up.
ARD is pretty weak compared to MS SCCM, and super-weak compared to something like AirWatch or Zenprise (MDMs).
It can easily create thin images for setting up new machines (make a small base install and then give the computer applications X Y and Z), package software and remotely deploy it to computers, create rules to push out new versions of software to computers, etc. You can set up self service so that users can browse a general store of applications but you can also assign applications to specific computers. They handle iOS devices now too, and their community is great. It also has SCCM and Altiris plugins, but I haven't used either of those features.
That being said, there's not a chance in hell I'd install so much stuff onto my mac.
IMO, Vagrant and VM's in general are what you should use to develop your web applications. (I happen to be opinionated about this :D )
Mainly this is because I believe in matching your production environment for development.
It's also because I'm anal about installing memory-using apps into my little MBA.
However, for things like Minecraft, onepassword, wget, sublime text 2 - this is really awesome.
It's pretty cool. And even if not everyone can put the time into building that sort of solution from scratch, pretty much everyone can ride their coattails
Boxen doesn't seem to care too much about what you want to install, so you could have it install Vagrant and VirtualBox for you.
Not really. What it gives you is easily configurable, absolute control over memory usage. I can ensure that my local dev environment doesn't use more than 2GB of RAM. From one setting. In one place. In about 20 seconds flat.
Perhaps this speaks to OS X's rather wimpy, squishily configurable control over memory usage.
I disagree. That doesn't belong in a consumer OS.
Bzzzt! Jumped to conclusions & placed words in my mouth! Thanks for playing. BTW, I'm an iOS developer and I've been using OS X as almost my exclusive dev environment since 2003.
Sorry, forgot that I have to keep quiet about shortcomings of OS X, or else.
> I disagree. That doesn't belong in a consumer OS.
I applaud your brave stance against the straw man you just conjured. [slow clap]
Enjoy the ride home on the short bus, man.
Also, doesn't admit or apologize when he's shown to be wrong, and responds with cheap insults instead.
On vagrant you have your production tool chain open in terminal (via ssh) where you can run a dev version of what you are building by forwarding ports to local host
You then can run browsers, IDE consoles across as many monitors as you want
*obviously web dev centric
I have many happy coworkers who use Macs, but please don't make me use one.
Note: I've been using macs daily, for hours from OS 6 to OSX 10.3, before I started using Windows and Linux more.
People who do some other stuff (heavy email users, some video/audio people) do sometimes use Windows (it's weird, but I think the Windows audio stuff is better for realtime now than Mac).
(the exceptions are people with FreeBSD, NetBSD, and Linux laptops)
I'd be pretty comfortable as a Silicon Valley employer only supporting Macs for office automation, and then either Mac or Linux for development workstations. If someone really wants Windows, s/he can support it independently.
The harder problem is phones -- there are people who are religiously attached to iOS and to Android, and you basically need to support both. There are pretty good MDM tools to cover both at the same time, but it does mean you can't push enterprise apps unless you do crossplatform development.
I'm not religious about platform choice, but I'd hate to see a future where developers are pidgeonholed with regards to their tools
Edit: currently 13 of 15 of our devs use Mac
A reasonable compromise is Mac laptop for office tasks, and a VM for development, and then a desktop development machine. If someone is super mobile as a developer, I could see a Linux laptop as an option.
The annoying thing is that if you really want security, you are basically stuck with Windows 7 or Windows 8 now, at least for desktop/laptops, and iOS or BB for phones. (Windows and platform management has gotten better -- OS X is actually the least secure OS in a major corporate environment today, due to lack of security and management tools. It's still decent for unmanaged use vs. Windows or Linux.)
In an environment processing highly sensitive information (say, a law firm working on M&As, or a print shop handling annual reports), where the tools aren't that essential to work, you could have a legitimate "you must use only our locked down systems" argument. I wouldn't really want to work in a place like that, though.
The long-term solution in high security environments is probably a mobile-based OS for desktop/tablet/mobile use, and then virtual desktop into either a super locked down existing desktop OS, or some new environment. For a lot of stuff, locked-down tablet/mobile (or ChromeOS) connecting to SaaS apps could probably do it.
I had to babysit OS X all the time, due to its lack of a good package manager. This was years ago, I don't know if it has improved, but it grated me greatly.
Well of course, OS X has heavily improved since the early days (I'd call 'early' everything below 10.4). The surge of developers wanting to use OS X has also increased the demand for package managers, and so they came. First fink, then macports and nowadays most use homebrew. You install them once using the standard OS X pkg installation facility and then you're good to go - the range of 'backend software' packages in brew is comparable to apt-get I'd say. For everything from linux that needs a GUI I have a VM, for everything Windows-only I have another VM. There's never even a question whether I can run something locally - once you have that it's hard to give it up again really.
That being said, the best OS X release is probably 10.6, since then I don't like the direction very much - but on macs you're basically forced to use the newest OS (XCode compatibility, hardware compatibility once you upgrade your macbook). The iOSification hasn't been a dealbreaker to me so far, it's IMO still a better all round experience than any other notebook, especially considering the service quality, which is bar none where I live. Friend of mine bought a $2.5k lenovo - the board went dead after 1 month, they picked it up and he hasn't seen it since the last six weeks, no replacement. I had a similar issue with my rMBP - got it back, fixed, after 2 days. Similar stories about Dell. HP might be better, but their hardware is crap IMO. I just can't trust any other laptop manufacturer at the moment, which makes me sad.
Fink was only out of date because they weren't keeping it up-to-date (perhaps their builds weren't automated? That's pure speculation, I have no idea). Debian manages to keep things more or less up-to-date with more packages (and architectures) than Fink ever had.
Incidentally, if you put your Homebrew in /usr/local then it will often install using the "pour" technique which is pure binary distribution.
I loved Fink until it started languishing, and I love homebrew now: its "everything in git" philosophy and very-open-to-pull-request attitude of the maintainer make me optimistic that it won't slow down and become irrelevant like Fink did. I think that's the real difference between the projects.
Small-scale? Run a simple script. Large-scale? Use a network-hosted configuration (optimally read-only root and network booting so the entire system is known-good) to avoid the entire class of configuration drift / migration / state-accrual problems associated with the above.
I just see puppet as sort of trying to provide the latter and failing, resulting in a complex version of the former.
If they are often on the network, you can use occasional rsync, for example to the system partition either at boot or in the background. That way, boot is possible even if disconnected.
If they're not often on the network, and you are seriously considering doing this at all, I would reconsider which problem you are trying to solve.
Also: I don't know about anyone else here, but I would prefer not having a company manage my laptop. The use cases for such a scenario, in my view, are pretty hard to imagine.
The feeling that I want to write something quick or easy shouldn't depend on the size of my infrastructure, it should be easy either way -- 500 servers, 50, 1, doesn't matter.
That being said, our modules do need some patch-love for things like launchd and OS X groups, we've been a lot more Linux/Unix focused to date. (Patches are welcome of course!)
The resource model is still very puppet-like (state=absent/present, idempotence, etc), in that we have states and convergence, but I have tried in many places to not make this a programming language, and keep the amount of syntax down. A key focus was making the content readable, so you didn't have to think about dependency ordering and could tell by skimming the playbooks top to bottom exactly what they will do.
There are two things I want:
1. To describe the desired correct configuration as a directed, acyclic graph.
2. To have some automatic compare-and-repair mechanism regularly bring my systems to such a state.
If you don't have a good tool for those two requirements, you wind up reinventing it anyway.
Your small shell scripts start being littered with lots of checks for this and that file, if-thens and cases. Then they break when script 27 silently gets out of sync with script 42.
You realise one day that system configurations can be expressed as DAGs (possibly it's a new insight, perhaps you were reading ITIL documentation). And you begin to dream about a tool that can take a descriptive DAG and generate the correct shell scripts. You have now half reinvented puppet.
But the systems still get out of sync. So you start tinkering with a tool that periodically checks each system and reruns the correct script. Now you have to ensure that your scripts are all idempotent. All those if-thens and checks creep back in.
So now you begin to dream about a system that only generates the steps needed to close the gap between the DAG and the current state of the system.
Congratulations. You just reinvented the other half of puppet.
If you have already decided on your solution before you have analyzed the problem then there's nothing to discuss.
> 2. To have some automatic compare-and-repair mechanism regularly bring my systems to such a state.
Right. My fundamental point is that puppet-esque solutions only describe certain aspects of the system state, ultimately failing here and resulting in 'configuration drift'. Entire system images are far more elegant.
Most of the use cases I have seen for puppet-style stuff are legacy-situation based and only make sense within that context.
So what is the alternative to puppet-style solutions? Personally I use full system images on cloud and cluster solutions (corosync/pacemaker) to maintain server state.
You can see how I arrived at that solution.
> My fundamental point is that puppet-esque solutions only describe certain aspects of the system state, ultimately failing here and resulting in 'configuration drift'.
Particularly with runtime / process supervision. One of my pet peeves.
> Entire system images are far more elegant.
Yes and no. It really depends on what your cost/benefit tradeoffs are. I like system images for startup, but systems still drift from their initial configuration no matter how that configuration is established (DAG or blob).
Even with system images you'll need to detect divergence from ... what, exactly? Doing a byte-for-byte comparison is going to suck.
Do you just kill and relaunch periodically? I can see that being stochastically effective.
Personally no, but you easily could.
> It really depends on what your cost/benefit tradeoffs are.
Well, realistically, to facilitate automated deployment and testing of multi-machine systems you really do need to be operating at this level of abstraction. (ie. to declare your desired state). Once you reach this point, a configuration file for some version of a package within some node is really a distraction rather than a help; it should have been abstracted to some known-good state and taken out of the equation if you are to have any hope of staying sane. Many people use entire system images coupled with service monitoring as the de-facto segregation point for management purposes.
> Do you just kill and relaunch periodically? I can see that being stochastically effective.
While that's entirely possible and probably a reasonable approach in some cases, the corosync/pacemaker de-facto/spiritual approach is to detect issues with a given resource (roughly: 'service instance'), automatically destroy it and fail over to another instance thereof, and then potentially start another instance to replace it automatically on the same, or some other cluster node. To facilitate rapid failover, the master/slave paradigm allows you to have live backup nodes running and promote them as masters easily. Any type of hardware or software can be scripted as a managed resource. The type and frequency of resource health monitoring checks can be custom defined.
I still feel you're ultimately defining a DAG ("I want 20 web servers, 3 database servers and 2 load balancers") and relying on some compare-and-repair (health checks and replacement).
But you've moved your unit of management from packages etc to machines. I've previously argued that this is the key thing that will change web hosting economics. I was sorta wrong, but I can see now where you were coming from.
In an interview with MTV on his new formal collection, taglined 'S-eXpression', Nerd-DAGgy was asked what the secret of the hot new look was. In characteristic brevity, he replied "It's declarative."
-- This Season's Assembly, Fashion World, February 16, 2013.
Puppet-like systems have the same issue in that they don't attempt to specify the entire system, but at least with puppet if there is an issue due to system drift you can start with a clean base installation and re-run it.
You just uniquely name the environment, for example with a version number, and/or use a snapshot-capable datastore.
I guess you could just write a changelog, but that requires a large amount of self-discipline to ensure that the changelog exactly matches system state.
Definitely don't do this.
Let's say you discover some minor instability, and discover that it was introduced 8 months ago, but the changelog for that image was completely innocuous. Can you do anything other than throw out all 8 months' worth of images and try to work your way back to present via the changelogs?
If you write a test that can trigger the issue and replay the test against past images ("regression test") then you will identify exactly where the issue was introduced. The key thing is: don't have manually configured what-not in production. Keep it versioned, keep it solid, keep it known.
Puppet-like systems have the same issue...
Exactly. This is my point. They don't really deliver on the promise of decent automation, because they are inherently patchwork/partial-scope in approach.
The other aspect is that many admins are ally do not like the bad habits associated with an image-level of abstraction, which leads to many hidden dependencies and configurations settings. Aa package-level manifest like Puppet, Chef, or Ansible enables much more of a complete explicit specification of the environment along with cross-machine dependencies. Such a description also allows easier reassembly of subsets of the system into different combinations.
The trend towards image-level abstractions does work for many as an alternative: Netflix for example avoids Puppet and just reassembles AMI images on each deploy, with auto discovery of cross machine dependencies at runtime via (eg) Zookeeper or other cluster facilities. But they have the discipline not to get into an image sprawl situation.
I find OS X to be pretty frustrating to work with natively. I've long tried to keep things like Postgres and Python consistent, but even a 10.x.y update can break things. Even with homebrew and postgresapp, it's challenging to keep the system from breaking. I'm currently running 10.7 precisely because I didn't want to rebuild my setup on 10.8 (I've since started using Vagrant much more aggressively to help solve this).
I realize that part of the point of boxen is to help set a baseline but it seems like a Vagrant box that mimics the actual Github stack would make more sense.
I would be interested to hear more about why they made that choice.
It was this line specifically which implied that Boxen is used to emulate the production environment in OS X and spurred my comment.
[1] https://puppetlabs.com/puppet/related-projects/dashboard/
[2] http://bitcube.co.uk/content/puppet-dashboard-six-months
There's a lot more to this than just setup scripts, but if all you need is setup scripts with the occasional update, then by all means just stick with that. Our needs just happen to be beyond that.
Disclosure: I'm a big pivotal_workstation and Chef fan, but excited to see what new angles boxen demonstrates :)
Their our-boxen template[1] installs homebrew for you as well as some other software.
Lots of good ideas in there (some of which inspired early work on dotCloud, including https://github.com/dotcloud/cloudlets )
Calgon, take me away.
Now the question is: can we get this for Linux users?
https://github.com/seryl/kindness
supports updating itself and it's templates form the git repo it's referencing.
Either way it seems pretty cool.
I learnt this when I considered starting a linux-optimised PC business back in the early noughties.