matlock

Do you want to test your ansible scripts during deployment, or do you just want to run ansible deployment after the tests are run?

The basic workflow that we at Codeship tell people is test each repository by itself and if that works push to a staging environment.

Once the push into the staging environment is done you restart the last build on an integration test repository that will thoroughly integration test the whole staging system.

When you build service oriented architecture some backwards compatibility for the time that systems are updated are in our opinion the best way. If there are breaking changes treat every part of the infrastructure as a separate api with specific guarantees and just build a v2 api so you can update the clients, but the old ones still work for a while until you can remove the old clients.

Continuous Deployment was also always part of it. Basically we upped from 50 free builds to 100 and announced it properly. But the basic feature set hasn't changed and you currently get all of the features in the free plan that you would get in a larger plan.

Codeship - Boston, MA or Vienna, Austria

At Codeship we build a continuous Deployment service. Our mission is to make software teams more productive by helping them to release early, often and safe. We're building for the builders.

We've been in production release for several years, have thousands of developers using the service and are growing rapidly. We've closed our Series A earlier this year.

We're looking for Rails Developers and SysAdmins/DevOps who love building tools for other developers.

Read more about us on our Jobs Page: https://codeship.io/jobs And how we work in our Blogposts: http://blog.codeship.io/category/the-codeship-workflow

Send me an email to flo@codeship.io. You'll like it here!

I think this shows a different understanding than for example we follow at Codeship.

Tests aren't there to test our code, but to make our Users happy. e2e tests make sure we don't break features and are able to validate new features don't kill the most common workflows. Thus we can innovate quickly, build new features and refactor. This makes our users happy. Unit tests, controller tests or others are mainly an optimisation as writing an e2e test for every failure state is not an option.

It's a different understanding of why we test, but makes sure we focus on the right things with it.

at Codeship (I'm one of the founders) you can run your tests and use our app engine integration to push whenever your tests pass. Setup takes a minute. let me know if you have any questions at flo@codeship.io

At least some parts of this post touch on immutable infrastructure, basically just replacing faulty systems and rebuilding them from scratch everytime you need to change it. Relatively easy with AWS and Packer (or other cloud providers) and super powerful. I've written about this a while ago on our blog: http://blog.codeship.io/2013/09/06/the-codeship-workflow-par...

Niiiiiiiice. Da ist der Wiener Humor ja auch noch in den Kommentaren. :)

>Bavarian German

you wouldn't want to say that in Vienna though :). We are pretty proud of our Austrian German. Winters are cold, and summers are hot, but still fine. Really nice to live here though.

Ever plan on visiting Vienna? Why not live there and build tools for other developers. We are hiring at Codeship (https://www.codeship.io) for our Vienna office.

And if you ever come visit the city let me know at flo@codeship.io. Happy to show the HN crowd around town.

https://www.codeship.io - Working on bringing Continuous Deployment to every team, because it's so much better, but still hard to accomplish for small or medium sized teams

Agreed, we've never done a rollback as well, but having the automation in place makes it easy if necessary. Automating still should be triggered by a manual process.

We would probably get a good amount of better performance out of it, but our current Heroku bill is low enough that even cutting it in half would be a small thing compared to changing the ruby version every developer uses.

While technically interesting, it just doesn't make any sense for us from a business perspective to put time into it.

I bet it would be a lot faster though. At least it is according to all the performance measurements I've seen in the past.

Yes it even works very well on MRI. Definitely a good choice

Continuous Deployment and Delivery both are there to remove the fear of change. Coupled with testing this is definitely a way to 10x your engineering productivity.

We've stopped counting how many times per day we push changes at Codeship as the number is meaningless in itself. If you feel you can't push regularly you giving up before starting the fight already.

It's just a very different and way more relaxed method of software development where you can actually focus on pushing the product further without getting stopped by infrastructure all the time.

Immutable Infrastructure will be the next big iteration in getting this 10x productivity I think.

While there can be analysis paralysis having a clearly defined workflow and writing a test for that flow before you even implemented that feature worked great for us.

Writing a functional test at this point helps in understanding the problem space and interaction with the service quite well. And with the functional test in place it is a lot easier to see which other part of the new feature needs to have unit tests in place to make it very stable.

At least that has worked very well for us for a long time now

In my opinion it is. You should start writing functional tests that test the application from the users perspective. Capybara/Selenium/Cucumber for example are a nice combination for this.

Unit tests are great for catching specific small issues, but you always want to make sure that your users can go through the most important steps in your application. These need to be thoroughly tested.

The most important thing with testing is getting started. Over time you can build a better test infrastructure and tooling, but just start with it.

At Codeship we focus on functional tests first. We use Cucumber/Capybara/Selenium a lot to test the user facing functionality. This way we can be sure that the feature works on the highest level. For some parts you might need to go down to unit tests, but start with functional tests first.

If you want to get started with testing your system try the following:

Everyone in your team writes down his 7 most important workflows in the application from a users perspective and ranks them. Then put all of the workflows together and try to find the 7 most important ones your team agrees on. Then find a tool that helps you test those from a users perspective. Build the whole toolchain (Tetsing tools on every developer machine, Continuous Integration server/service, ...) so adding new tests is trivial.

Now there is no more excuse not to write tests.

For mocking take a look at a screencast we did a while ago:

http://blog.codeship.io/2013/06/11/testing-tuesday-9-stubbin...

But still the most important thing with testing is getting started and having the whole workflow in place. Even if there is just one test, getting to a point where it is easy to add new tests needs to be priority #1

I can only follow on arohner. We discuss business with all of our developers all the time. Totally depends on the people, but in our team we specifically select for people who see the bigger picture and want to talk business as well.

Especially the first hires need to understand all parts of the business, not just their specifics

Jenkins has done a great job in getting continuous integration as a workflow into the hands of many developers.

But their usability is completely out of date and their current user base doesn't let them evolve to where they would need to be. Especially integration with PaaS services like Heroku should just be there from the start, but has never been a focus for the Jenkins community which still seems to be very on-premise based with all of their infrastructure.

More and more developers seem to be jumping on hosted solutions now (e.g. https://www.codeship.io where I am one of the founders)

We've published two blogposts how we sped up our test and boot times considerably. Maybe helpful to some:

http://blog.codeship.io/2013/08/21/faster-test-suite-boot-ti...

http://blog.codeship.io/2012/11/15/speeding-up-our-test-suit...

We improved our Go support on Codeship (https://codeship.io, I am one of the founders) over the last couple of days. Give it a try and if you have questions let me know.

But that's already an investment that can get you quite some resources on AWS for a while.

But then you are probably getting close in pricing to running the stack on AWS, as you don't need to have that much reserved infrastructure when you can easily replace it when there are problems with the server

It will only charge the hourly reserved costs.

From: http://aws.amazon.com/ec2/reserved-instances/

Easy to Use: Reserved Instances are easy to use and require no change to how you use EC2. When computing your bill, our system will automatically apply Reserved Instance rates first to minimize your costs. An instance hour will only be charged at the On-Demand rate when your total quantity of instances running that hour exceeds the number of applicable Reserved Instances you own.

>EC2 lets you roll a globally distributed solution with good tooling for low cost

This is the Crux. There are other options available, but going cloud makes it very easy to roll out something big without the necessity to have experts in all the lower level parts of your infrastructure. It lets you focus and the price for that is absolutely reasonable

But even in that case there is a single point of failure in the host. That's all manageable, but it is effort, that could otherwise be put into developing the application

Reserved instances are not bound to any specific VM. When you start a virtual machine and there are reserved instance slots free it will charge you the reserved instance amount. If you provision more machines it will charge the standard amount.

>A smart company will do a cost-benefit analysis, not blindly go to the cloud

But often times those cost-benefit analysis don't take into account how quickly you can improve and work on your infrastructure. The performance or cost improvements need to be a lot to slow down your team even a bit.

Especially for startups this can hit them very hard.

I absolutely agree, but oftentimes small cost optimisations are done by teams without thinking about how this impacts the speed with which they can build their business.

The cloud is not the definitive answer for all teams, but leaving all the power cloud computing gives for a little cheaper hosting at the beginning of a project seems wrong to me.

Thanks, going immutable with all of our infrastructure made our system way more stable and productive