Hacker News Reader

I'll be downvoted by the "newer generation" but here's a pet peeve of mine

The same reason "why we tag shit" is the reason downloading packages from gem/pip is not enough

Who guarantees that when you do your deploy, the library you need is there? How many times did your build break because it had a glitch downloading the package?

Keep a way of rebuilding and deploying your software. All of it

> How many times did your build break because it had a glitch downloading the package?

This happens very infrequently in my experience, but it's simple on most CI servers to restart a build.

I'm pretty sure that wasn't the point - however infrequently it occurs, it violates the principle that only changes introduced since the previous build could have made it fail.

In practical terms, this means that you never know whether change X is faulty or simply the world is temporarily faulty.

(Of course, this may or may not be not useful to you to know.)

True, but a build could fail arbitrarily fail for any unexpected reason. Pragmatism is important, I'd avoid adding an inconvenient step to my build/deploy process just to mitigate an infrequent edge case.

Infrequent (I don't remeber that it happened to me during the hundred or thousand deploy I made with capistrano) and mostly harmless.

If the deploy fail for any reason, it's aborted, you just have to run it again.

What do you do if the package servers go down for a couple of days? "Oops! Guess I can't deploy. Oh well!"

Maybe my brain is wired differently, but having to tell a new hire "Oh yeah sometimes the build randomly fails and we just press retry until it works" would just feel kinda embarrasing to me.

That's not what I said. The point is one might as well say it never happens for how often it happens in practise. I have only had it happen once in recent memory, and even then it was because the hosted CI we use was experiencing issues.

> a build could fail arbitrarily fail for any unexpected reason.

Good engineering is the attempt to prove that statement wrong.

I also wouldn't normally expect a problem here, but this actually happened to me yesterday. Some 3rd-order transitive dependency resulted in trying to download a file from https://rubygems.org, rather than http. Apparently my SSL certs were out of date (on both dev and prod) and so I had to spend a few hours Googling how to make RVM + Ruby Gems update my certs.

It's really easy to write these kinds of things off as rare occurrences, but this is exactly the kind of thing that causes us to talk about "fear" when deploying.

Edit: solution was to run 'rvm osx-ssl-certs update all' on OSX, which is unsettling, as it installs certs all over the disk. Then on Ubuntu it was "yum update openssl", I believe.

I've got this problem on a machine at the moment, what was the solution?

Nitpicking: Yum on ubuntu us highly unlikely (it's using .deb packages, apt-get). That's what you'd use on a rpm based distribution (Fedora, CentOS).

You're correct, it was apt-get. I also have a Centos image around, so I have yum on the brain.

If it happens even one time while you are in the middle of trying to resolve some "emergency", then it has happened one time too many. Customers don't care that you weren't thoughtful enough to keep a mirror of everything you were using.

I remember my first brush with this when Maven first started getting popular. Apache had to move their servers to a new datacenter and the server that had the drives for the main maven repo was lost in transit. It took five days to bring everything back up. During that whole time, nobody could build anything unless they had local copies.

Ever since then, I'm pathological about ensuring that I have a local mirror of all my dependencies.

Mmm. I inherited a rather hardcore strain of liking entirely deterministic builds from being a Debian Developer and would never give up the dependability of it now. In fact, the thought of a build process connecting to the internet to download code just seems so utterly broken in my world view I actually find it difficult to explain it properly to others.

I'd love to be able to do it without embedding code copies, or at least have a smarter system than that.

> I'd love to be able to do it without embedding code copies, or at least have a smarter system than that.

Maybe a section on the Server where gems lie, and a section on the hard-drive of your dev computer where gems lie - and then in a build process, it finds the gems with a certain checksum or tag or something similar. That's an interesting idea you have.

Sounds like you might like https://github.com/heavenlyhash/mdm ... yummy hashes for determinism, good for binary blobs, embeds semantic links in git, and uses git transports so you can swap out network or local filesystem trivially.

But for the most part these build processes are for web apps... If your build can't connect to the internet, your web app isn't going to work (even if it can build).

Sure, but it's not like the entire Internet is up at once. If rubygems.org goes off-line, it shouldn't mean you can't deploy your Rails app.

Why does my personal (or intranet) webapp needs an internet connection to work?

Actually, your web server being able to make requests out on the open internet is considered a security liability.

If it lives behind a proxy, a web server being able to talk to anything but the proxy server and database is a matter of developer convenience.

for High Availability you would be a fool not to either run your own mirror, or package everything you need into a single bundle.

This. People seem terrified of running their own gem or apt repositories to mirror their dependencies - or even to host their own code.

The dirty little secret is that these things are trivial to set up. Anyone whose build or deploy has been broken by github, rubygems.org, ruby-lang.org or even rvm.io falling over is Doing It Wrong. While I empathise, I have very little sympathy for people panicking when github gets DDoSed again.

related - I've used this for running my own Gem server for years - It really is trivial: https://github.com/geminabox/geminabox

It can be even simpler. I use Apache serving static files out of /var/www/gems, and a cron job to rebuild the index every 5 minutes if anything's changed. Caching gems is just `rsync -avz $GEM_HOME/cache <server>:/var/www/gems/`. Publishing internal gems I've built is just scp.

Geminabox is simple until you want to run with a lot (500+) of gems. And then index-building is horrible - it basically never finishes.

The complexity comes in mirroring the right set of other people's gems, which match the exact combined set of all versions used in all your various projects.

There is not a good "pass-through" gem server that just automatically mirrors what it is asked for from RubyGems. There really, really should be.

Yeah, I'm right there with you.

Rubygems is a wonderful and powerful tool for getting libraries set up and adding new code. It's not really suited for putting things into production, especially when people think pulling old versions (for whatever reason) is acceptable behavior.

At work we install and set up libraries on development machines using bundler/rubygems, then go in and version lock everything after we know we'll be using it. Building a release-worthy version involves repackaging all of the gems as rpms so they can be installed along with all of our other software when building the new (virtual) server.

It's a pain in the ass, yes, but we have never had library issues crop up in production. I know exactly what is on the system, and fixing anything is an obvious and straightforward process.

You don't have to make things this formal/official, but it's pretty easy to achieve library version updates and know that what's on your server will work correctly at the same time.

I'm about a third of the way through writing exactly that, covering deployment (capistrano) and basic server provisioning (chef), specifically for Rails apps. If you (or anyone) is interested my email/ twitters in my profile. Aiming to have it out end of 2013/ start of 2014 but would be happy to release it to a few people early for feedback.

Capistrano, what a Byzantine way to deploy software; I haven't typed cap deploy in about 3 years. Heroku's model of Git-based deployment and 12-factor apps has never given me a problem. Also Heroku deploys are de facto tagged as Releases (https://devcenter.heroku.com/articles/releases). I can't wait for Flynn (http://flynn.io/) to be released, it provides a Heroku-esque Git-based deployment strategy on any host.

P.S. I enjoy reading your comments as do many others, but I feel you hand-wave over important points sometimes, e.g. Why you feel tagging is important. If it truly is a 500 word discussion it shouldn't be hard to synopsise.

It is tangential to the point of the comment, since this only gets you to 0.001% of a Rails deployment which works, but if you're interested:

Tags, plus sufficiently advanced dependency management and database migrations (n.b. not trivial!), allow you to track and reproduce past states of the deployed software.

For example: A customer reports a problem. You verify the problem exists. The customer reports that the system didn't have the problem last week Tuesday. If you want to know why, tagged releases (and a record of them -- incidentally, another one of the 643 things to know is "All deploys should be recorded somewhere") let you travel in time back to the software in use last Tuesday, verify if the customer's recollection is accurate (well, sorta), and then compare just what has changed rather than trying to run down the bug without any historical context.

Naming things also lets you talk about them, which is helpful, because you'll want to talk about individual releases of the software. (I mean, sure, you could use random hexadecimal numbers, but for whatever reason people find production_release_20131007_3 a bit more informative than 066630de00d242137efab6bf21b8ea04aeee7a1d. One glance at the first one tells you that it's the 3rd deploy from October 7th, assuming you've used the company's naming convention for more than a minute. The second one tells you nothing and is difficult to pull useful information out of without having to manually cross-tabulate it with you git repository, which would be unfortunate if it were e.g. attached to a log file, customer support request, or Wiki article about known-good releases for interacting with external systems.)

Thanks it's not for my benefit, it's for the engineers that might not know better and blindly follow something you say because of your standing in this community. Cargo-culting is not a good habit to encourage in software engineering.

FWIW dates in tags are redundant because the tag itself has a time stamp. Re: environment names in tags, IMO they don't belong in the repo.

Thanks for replying.

> FWIW dates in tags are redundant because the tag itself has a time stamp.

Having them in the tagname makes it much easier and more visible.

Times are in the git log. Same as the commit message/tag name/etc.

It's still much easier to scroll down a list of tags and find the right one by date than to look through the git log and look at both a tag name and timestamp.

Personally I find git log easier to handle than an enormous list of tags.

"Good judgement comes from experience; experience comes from bad judgement."

This is where triggering deploys through Jenkins is really helpful. Now you have a record of which SHA was deployed, when it was deployed and a copy of the console log for the deploy. Can't do much better than that.

  after "deploy:restart", "deploy:tag_release"

  desc 'Tag release'
  task :tag_release do
    `git checkout #{rails_env}`
    `git tag #{rails_env}_#{DateTime.now.strftime "%Y_%m_%d-%H_%M"}`
    `git push --tags`
    `git checkout master`
  end

Boom!

That's not going to match the release name that capistrano chose, right? Seems like you'd want to use latest_release or something.

Yeah, modify to taste. I only wanted to post some example code to show how easy it is to automate tagging.

My code won't be appropriate if you don't use master/staging/production branches, either.

We do the inverse. We tag first, then instruct Capistrano to grab the latest release (with a prompt). We have a release plan though, where software goes through QA before it's tagged. Only software ready for production gets a tag. Everything up to that point is done on a release branch.

    set :branch do
      default_tag = `git tag`.split("\n").last

      tag = Capistrano::CLI.ui.ask "Tag to deploy (make sure to push the tag first): [#{default_tag}] "
      tag = default_tag if tag.empty?
      tag
    end

Just use git flow for releases. For non-trivial deploys, you may want to check out teamcity.

> Capistrano, what a Byzantine way to deploy software...

Why would you come here to say that? Capistrano is one way to deploy software. Heroku's method is another. Neither is wrong.

This kind of thing is why author's of open source tools like Capistrano get burnt out and just want to run for the hills. Because people with no skin in the game have to come along and shoot their mouth off.

You're probably a really nice person, but when you do things like this, you hurt other people and you make yourself look like an asshole.

Well-said. Heroku-style deploys are not a replacement for Capistrano-style deploys, merely an alternative. Capistrano is a much lower-level tool. I use Capistrano also for general sysadmin tasks that comprise multiple servers.

Is it really Byzantine? Never ever had this thought. Once its setup, you just cap deploy - how is this much different from git push heroku? To do anything complex with PaaS stuff you still have to write code to setup the environment - likely no more than for cap.

There definitely is a product here, and it's already being built. Docker is the underlying framework for which we'll see significant improvement for deploys in the near future.

In the meantime, and if you're on AWS, check out OpsWorks. It's the cat's pajamas. It both makes much of the hassle of deploying a thing of the past, is extremely flexible, and makes it possible to have Heroku-like "git push" deploys without giving up control (or being forced into Beanstalk's opinionated ways). Amazon made a very smart purchase when they bought Scalarium.

you don't happen to have a good online reference on Why We Tag Shit, do you?

Not exactly a product, nor wholly Capistrano-centric, but I recently put together an article on how I deploy Rails on a new VM from scratch: http://robmclarty.com/blog/how-to-deploy-a-rails-4-app-with-...

This is one of the biggest gotchas of (some of) the high productivity full stack frameworks that are out there, in my experience. I've also had issues getting Meteor deployed, and while it's in many ways a smaller, simpler beast than Rails, it has the same kind of black magic and hitting-a-wall-at-100mph deployment experience.

Both Rails and Meteor will let you develop apps at breakneck speed, with their powerful, magic cores and their excellent module ecosystems, but when you get to deploying them you'll sometimes wish you'd stayed on LAMP or (god forbid) ASP.NET. Yes, I really said that.

I glad you said that. I'm seriously considering moving from RoR to ASP.NET/C# platform.

We're spending $50k-$100k/yr on Linux system administration of all the small and big moving parts that always fall apart and requires constant babysitting.

I want to give MSFT all this money and use MSVS, C#, ASP.NET and Azure for everything. I love MSVS and always loves MSVC (back in the days). I love performance of C{++|#}.

I don't feel inspired to build great SaaS solutions with shitty interpreted languages, forcing to use made-in-the-basement plugins (or gems) and all that trailing crap of sluggishness, upgrades, fixes and weird behaviors that follows.

Now you all may downvote me!