Faster Web Development With Virtual Machines at deviantART | Hacker News Reader

Faster Web Development With Virtual Machines at deviantART

dt.deviantart.com

65 pointskemayo45 comments

Disclaimer: I am the creator of what I'm about to plug here.

If anyone is looking for an easy, low-risk, documented method of beginning development in VMs, check out Vagrant[1], which is what I wrote this for. Creating dev VMs from scratch is pretty tedious and there are a lot of steps involved. Vagrant abstracts this away and provides a tool to automate all of this in a reproducible manner.

[1]: http://vagrantup.com

I love vagrant! It was super easy to create an archlinux base box and i've been using chef to automatically provision environments. Setting up the environment for a new project takes minutes and is both totally automated and separated from my actual OS. Thanks!

I started using vagrant last week to debug my Chef scripts and it is indeed awesome. Vagrant uses Chef (chef-solo) configure your VM from bare image to working server.

I see that is has a plug in system for other provision systems like Puppet, but the chef support works great out of the box.

I was hoping to contribute to the project by integrating fog the ruby library which talks to many cloud systems including EC2 so someone could build an image via Chef or Puppet, dev and test in their image, then provision and upload to the cloud. Could probably do this without integrating with vagrant by building a clever rake or capistrano script.

After talking with a few small PHP teams this seems like the logical conclusion whenever growing a team of PHP developers. Its just very hard to perfectly recreate your server environment locally on multiple OS's.

Its a great solution, but I much prefer working with RoR or Python/Django where recreating the server environment locally is much easier.

even for RoR or python - might make sense for those working on multiple projects simultaneously, using different generations of frameworks

There's RVM in the Ruby world. Anything similar for Python?

virtualenv

I'd say that seems to cover both bases for not needing a VM.

Every language should have something as simple as these. Heck, it should be part of the OS, IMO; why I can't (easily) run two different versions of Firefox with two different profiles at the same time is beyond comprehension.

A VM is a constant time investment, doing it for every language is linear with the number of languages.

Unless there's some serious increases in clock speeds, I'm not sure doing it for every language is worth it. As it is, VMs are a great, general purpose system that are easy to set up, use, and replicate.

You do need a VM if you use memcached/redis/some other thing and don't want to install it on your machine. Other than that (for 90% of cases), virtualenv has you covered.

It's been a while since I've done this; but:

* Install your different Firefox versions in different locations (so one doesn't overwrite the other)

* Use "firefox -ProfileManager" to create at least one different 'profile', which is Firefox's user data set

* Use "firefox -P <profilename> -no-remote" to launch Firefox. -no-remote allows multiple Firefox instances to run on the same machine. -P profilename is necessary because you need a separate profile for each running instance.

More info at http://kb.mozillazine.org/Command_line_arguments

Ooh, I'll have to try that. I've been using different profiles, but didn't know about the last step.

Still; mindboggling. Different version = different binary launched, which accesses a different and non-locked profile. Why the hell does it instead open a new window of the currently-running version?

As a heads-up to anyone considering this: Sync doesn't (currently) like multiple versions. 3.6.10 won't sync now that I've hooked up some 4 betas.

Bundler was supposed to remedy dependency hell problems and I believe it eventually managed to do that. But boy can I tell you stories about migrating from pre-1.0 bundler versions.

RVM is just great and simple.

What I’m trying to say though: Having done quite some side projects with node, the ruby stuff suddenly feels very enterprisey to me. More and more conventions might make it harder to keep track of everything in the future.

How is creating the local server environment on RoR or Python/Django easier than PHP/anything?

I've been developing locally for years and while I can't mimic the exact hardware of the live server, I wouldn't be able to do that regardless of stack I chose.

For the Python world, virtualenvs and pip requirement files make deploying a local copy of a project take about 30 seconds, and it's completely independent of your overall system and all other projects you happen to be working on.

Gotcha. With git and a Phing build script, I can do the same in PHP :)

As Pewpewarrows said, you can use pip with a virtualenv to deploy with one command. If your settings.py is any good, you can load your data dump in an sqlite database in one more command, and you're ready to go.

We used to develop historious like this, until we switched to redis/celery. Then it became too much hassle to install these in every machine, plus we wanted to test on the db we used (postgres) rather than sqlite, so we made a VM and now everything's dandy.

Your last sentence is an example of when I would like to vote "my displeasure" with a down arrow.

As others have mentioned, you make a bold claim, yet fail to give anything concrete as proof that it is "much easier".

I've found that developing against a VM is a great tool for a dev. For OpenStack swift, we develop all of our code against a VM. Since we deploy on debian and I develop on a Mac, I can use OS X tools to develop (TextMate, etc) and code against an environment similar to our production. After code works on a VM, we can move to a lab or a staging environment to do more scale testing before deploying to production.

Developing against a VM is not something I did before working at Rackspace, and I find the practice so useful that I will do what I can to continue doing it no matter where I work in the future.

This all VM thing is fine and dandy if you want to have an accessible mirror of your staging machine and reduce unnecessary stages. Although that in itself looks fishy - if developers want to stage something just to see if it works, then something is seriously wrong with your process or developers.

I'm amazed at the level of sophistication in this setup - using virtual machines and TCP proxies that do text replaces seems too much. They could have just built the configurability in the damn application. I have a .NET app that normally gets deployed on three servers and uses various remote resources like Amazon SQS queues, but I easily built a configuration abstraction similar to Rails' environments that lets me deploy on a single machine and even test locally with a debugger attached.

My point is that this problem looks as if it got solved at the wrong level of abstraction. Which, obviously, made it unnecessarily hard and complex. If I had to tackle it, I'd make sure developers do the right thing and try to make the app configurable enough first.

The mind boggles. There's probably a rockstar in there looking to put something on their CV/resume. Whatever the case, it smells wrong.

I only skimmed the article but I assume it’s about the benefits of running VMs to have solid dev environments with zero setup cost.

Let me ask another question: Will it become feasible to move code, tools and everything else into web services? Imagine you have your favorite editor (emacs, TM, vim) running in a browser window. Only that it’s live-updating like, e.g., Ethercodes or Wave. And you have very straight-forward access to SCM features like staging hunks of code. Also the code deploys straight to heroku or whatever. It could, of course, also go through closure or sproutcore in case of a fancy js client app.

Do you think it’ll all evaporate (I refuse to say cloud :) eventually?

The thing is I’m really going through a lot of pain setting up my system and other’s systems over and over again. Yes, there are ways to automate that but the effort is too high for most or we would all be doing it right now.

I remember Heroku had something like this a few years ago. It was pretty easy to get a taste of Rails.

I'm sure it will happen one day, but it's quite a big challenge and you're up against all those bad habits build over the last 20 years. For example, developers really like customization. It's quite hard to provide a hosted environment that is fully customizable.

Also, parenthesis mismatch in paragraph 3.

It's not clear to me why they don't just run development on their local machines. I'm not familiar with PHP, though. I'm all for having a build/test vm, though.

I good PHP environment that models the server is very hard to get going locally. At least on OS X. The only way I could get a good environment for Nginx+MySQL+PHP was to locally run a debian VM in VirtualBox.

Now this doesn't even take into account things like memcached that require a lot more customization.

Im confused, this post of yours seems a tad contradictory to your post about "setting up environments in RoR etc".

Developers aren't server administrators (usually). It's not the best use of their time.

I wish I had thought of this when we ran into issues with Windows people and Ruby. Too much of their time was spent fussing with stuff and crying about it. :)

“So now it should work, right?”

-“I’m getting an error”

“Which error?”

-“Unnamed method in [...]”

Then, a mountain of pain and torture later.

"But now it works, ok?”

-“Ok.”

---

Ever had to fix someone else’s system via Skype or phone? I personally believe writing a rails clone in brainfuck is more fun.

JRuby cn also be a good way to run ruby on windows.

I've had a setup like this going for years. I'm a big fan.

It's unlikely that your server is the same os as your desktop. So you waste time setting up versions of libraries and services for e.g. windows that don't always work quite the same as in production. Plus you can also take advantage of snapshots to mess around with your local vm, confident that you can always roll back to a known version. You can do simplified load testing without actually crashing your whole machine.

It also means your devs can use whatever OS they want - the hard work of setting up a dev environment is already done.

It's easier to give a new developer a VM image than have them set up Apache, 2 Apache modules, PHP, 3 PHP extensions, MySQL, memcached, ImageMagick, mplayer, a queueing system, a custom database daemon for statistics, and a Lucene-based search daemon. And all with the same options as production.

Any moderately complex app has a few moving parts. At least you have the web server and the DB. The DB has settings like the default character set, table types etc. These have to be kept in sync across all instances, dev, stage, and prod. The web server also has modules and a conf file with mod_rewrite rules which change and should be kept in sync. Sometimes you update a library or a version of ruby. Add a noSQL server, message bus, file storage, email, etc and you can see how it becomes a serious issue. And the problem grows exponentially the more developers there are each with one or more dev instances which have to get the latest changes and be kept up to date.

These things all have to be dealt with and should not be thrown over the wall and "let the sysadmin deal with it." That's where the value of the OS and infrastructure is a set of code and config files which are kept in source control next to our code and versioned with it is so valuable. Run "vagrant up" and a fresh consistant build every time. Then develop in php, python, perl, ruby or brainfuck if you want.

I do acknowledge that this is something that looks very "oh, how obvious!" to the modern crowd, but deviantART is a 10 year old PHP app...

Of course, even if you're running a modern web framework in Ruby/Python/somethingtrendy, a VM makes a lot of sense. It lets you run the exact versions of everything that your production environment is, without having to worry about it.

Developing in the exact same environment as you deploy in should be standard practice.

This reminds me of facebook development, where you need to have a live fqdn for all of your environments. We used to use a staging server and had a similar problem with huge amounts of small unnatural commits for things you would normally test locally. I ended up moving to using ssh tunnels to a server, where each dev has a subdomain redirecting to their dev box through the tunnel. This has the added benefit that you can work locally from anywhere you have wifi. Whether you use tunnels or VMs, both are a solid way to do development.

If commits were an issue before this, they should really have considered using git or something. You shouldn't aim for fewer commits, but less expensive commits, i.e. nothing goes over a network. Plus you can squash later if you really need to.

Anyway, VMs are cool but has anyone tried developing with NixOS? It seems even more suited for the tasks they want since it has rollbacks and so on: http://nixos.org/

Isn't the expense of frequent commits the cost of composing an intelligible commit message?

Great Article. As QA Analyst and Engineer, I know first hand about the pains of replicating production environment. Thanks for submission.

I do most of my work in python/django so virtualenv essentially solves much of these problems.

The problem I am running into now is that our production environment is starting to get complicated enough(mysql, redis, 2 different django projects) that testing all the components working together gets more and more difficult all the time.

Start scripting your environment, the more repeatable your deploy procedure is the better your deploys will be. The easier your deploys are the more frequently you do them, the more frequently you do them, the better your deploys will be. Rinse wash repeat.

Once you have your env scripted with EC2 you can setup / teardown environments easily for testing.

Having a VM you can trivially reimage is very helpful because you can test that (re)deploying packages to it is going to work, you didn't miss any dependencies that happen to be present on your desktop but not on your production servers, and uninstall won't leave cruft behind.

Another benefit of this approach is that it takes seconds to revert to a good state if you destroyed your dataset in a mishap. use snapshots or just restart VM from original image. all while using your preferred code editor (the app source is mounted from the host OS)

I wonder how long this took to set up? And why they don't run their production servers as VMs as well?

Edit: I think it's an awesome idea, btw, and something we try to push to web service shops at GridCentric.

Only use VMs or sandboxes on production-like machines.

Local development doesn't scale.

Reply on news.ycombinator.com