Ask YC: How do you upload to your web server? | Hacker News Reader

Ask YC: How do you upload to your web server?

29 pointssangguine46 comments

Hi. I was wondering how you upload your code from localhost to the web server. Right now, for my project, I upload the entire directory from localhost to my web hosting server and replace the existing files with the newly uploaded files. It was ok in the beginning because the files were small. Now that I have a bunch of libraries, it takes quiet a long time to upload all 600 files every time I upload.

I recently learned about rsync to upload only changes. I was wondering if you use this or other tools to upload.

View on HN

Git or SVN using Capistrano.

If you're deploying code, you really should be using Capistrano. Even if it is not a Rails project. It is a world of difference.

I was in the same boat as you not too long ago. As the others have said, Capistrano really is the way to go. Out of the box, it's meant to work with rails projects, but if you tweak it a bit, it'll work fine with really any language/environment. You can try www.capify.org to get started. Beware though, the majority of documentation floating around is for the 1.x release. By v2, a lot of the configuration specific details were changed though and unfortunately if you are starting now, the 2.x docs are really lacking. There is an effort underway to bring them up to speed, but so far I haven't seen anything. Read the docs from the 1.x release, as enough of it still applies to 2.x.

I found Capistrano a little too Rails-centric. I've had a lot of luck with a Python equivalent, Fabric. It works well with the Boto Amazon AWS library to manage EC2 instances. Capistrano has limited AWS support but it seemed immature at the time I tested it.

http://pypi.python.org/pypi/Fabric/

http://code.google.com/p/boto/

I use Capistrano and am quite happy with it, but even if you don't go that way, seriously look into some kind of deployment management software. Versioned deployments, simple and automatic rollbacks if something fails, and the ability to easy scale up deployments if you wind up breaking out into multiple servers for app/db/web.

It turns updating your server from a mild headache to one line that you actually enjoy typing in.

I use Capistrano too. It is written in ruby but can be used by any non-ruby project.

I use capistrano for PHP, Python and Ruby applications. Its simple and works.

After I'm done editing my pages, I save them to a 5.25" floppy and mail it via USPS to GoDaddy in Arizona with the instructions to place on my webserver.

Seriously man, it's 2008. Time to move to 3.5" floppies already.

Dude. He was being ironic.

Upvoted for the obviously intentional meta-irony.

I use Fabric - http://pypi.python.org/pypi/Fabric/ - with a series of build scripts. I can create and update several different types of EC2 servers (database, app, etc) with a single commands.

The update command does something like this:

* Check out all pertinent application files from the repository.

* Add any production specific files, for example build number or database setups

* Combine, compress, and number all Javascript and CSS files (combined.384.css can be cached forever, on update it becomes combined.385.css)

* Tar and gzip the production files

* Find all EC2 servers that need to be updated

* Upload the gzipped files

* Stop apache

* Remove old files on the server(s)

* Unzip the new files

* Start apache

This is one way to do it on Windows.

Use WinKey to map win-P to project.bat.

project.bat: rsync -Crltvz -e ssh /cygdrive/c/personal/project/ me@mydomain.com:/bigfiles/project

then I can just hit win-P to force the rsync. Runs in a couple seconds even with hundreds of files.

But my favorite if I don't need version control is to use <a href="http://www.sftpdrive.com/">sftpdrive</a> to mount the remote server as a local disk. It is $39, but is probably my favorite windows software. I've done the Samba thing and like this much better.

You could also try Novell's NetDrive. They don't offer it for download from their site, but I believe it's free.

http://www.novell.com/coolsolutions/qna/999.html

I use sshfs (sudo apt-get install sshfs) and mount my web server to a local directory. From there, you can do a variety of things.

Man - I'm low class apparently as I'm apparently the only one who uses Filezilla or just a plain FTP client. I support about 30 domains and this route helps me. I do have some internally-written apps that will upload files to a lot of different domains at once.

You should at least be using SFTP, since FTP isn't encrypted.

For my personal site, I use rsync, and I keep the command in a makefile so I don't have to remember it (I also use the makefile to generate the static pages from templates). It looks like this:

  sync:
  	rsync -az --progress ./ user@host:public_html/

  .PHONY: sync

But for anything significant, I use a VCS, currently svn or git.

You should really use ssh with that. Not much more complex:

rsync -azP local_dir -e ssh user@host.com:remote_dir

rsync is usually configured to use ssh by default. no need for -e.

Never knew that, but you're right!

Perhaps a bit noobish, but I use the web development software Coda (panic.com/coda) and it tracks changes for me; when I publish, it only uploads changed files since my last publishing.

i use mercurial, assuming your base directory is /www/pages (lighttpd), initially:

server: $ mkdir /www/pages/project $ cd /www/pages/project $ hg init

localhost: $ cd /path/to/your/project $ hg init $ hg add * $ hg commit -m "message" $ hg push ssh://usr:pwd@123.123.123.123//www/pages/project

afterward it's just the localhost (ignore "$ hg init")

Here's the bash script I use with Mercurial to deploy:

hg qpop -a ; hg pull ; hg update $1 ; hg qpush -a

The $1 is a branch or tag argument given to the Bash script. That way you can easily specify a particular version to update to (or rollback to!) on the command line.

The qpop and qpush will rebase any MQ patches you've made, e.g. local configuration changes, hacks, etc.

More on rebasing applied patches with Mercurial: http://hgbook.red-bean.com/hgbookch12.html#x16-28900012.8

I agree that you need git or hg or svn or /something/ of the sort for large sites, like sangguine has, but for small/tiny sites, other options are available.

Heck, I use sshfs to mount my blog as a directory & edit files directly. rsync has also worked nicely for me in the past. But then, these are all sites that I either don't care about source control, or don't have git/hg available on the shared server.

This is what I have set up. A Git repository on the server. I push all my code to the repo via ssh.

On the server there are Dev, Staging, and Production environments. Dev and Staging are bare checkouts from the git repo. I have deploy shell scripts that automate deploying to both (checking out from source control and running database migrations). The process should be that code is deployed to Dev, then Staging and finally to Production.

The Production environment is NOT a checkout from source control. It is an exact mirror of the Staging environment. And the deploy script to production does the following:

- tags source control with the new release number

- creates a tar.gz backup of the production directories

- does a database backup dump

- then uses rsync to copy files from the staging web directories to the production directories.

- lastly runs migration scripts on the prod database

I took a page from Rails and Capistrano deployment, and like the idea of versioning releases and using symbolic links. In essence, you update the latest code out of your source control system into /myapp/source, then you run a build and installation script which installs the latest copy of your code to /myapp/release-2008-05-22-114500 (timestamp), then makes a symbolic link from /myapp/current to the right version of the code. That way, if you release something broken, you can trivially roll back to a previous version. (It obviously doesn't always work for things like database schema changes.)

I actually have a commit hook in git that will update a copy of my website when I commit something with the right message. It's kind of crude, but it manages the busywork for me quite nicely.

My webserver runs Puppet and periodically pulls both the code and its own configuration from my Git repository (which I'm currently hosting on Github, but I may someday self-host).

Yes, I'm going against the grain by using Puppet instead of Capistrano. All the Rails people are probably going to crucify me, but I personally think this setup works quite well.

We use BZR for version control when we have a rep we arehappy to deploy

bzr push to a REP on the server login over ssh on the server bzr export REP run the test suite on the server (last check) cp relevant files to PROD restart the server (FastCGI)

everything automated and sequential one failure -> process stops completely (never happened so far)

A separate SVN checkout (with Tortoise), then WinSCP. Surprisingly, you can use WinSCP very well with scripting. Could also make a backup in the same script but didn't yet. The same setup for getting down the database from the production to the development machines.

It's not very professional, but for our needs it's perfect.

Are you using a repository to store your code? If so, you can check it out from the server and then copy it over.

Yes. I am using SVN to collaborate with my friends. How do you copy easily?

Commit your changes on your local machine then SSH to your server and do an svn update.

one problem with this method, is that your web directory will have all of the .svn folders sprinkled throughout your subfolders. svn export avoids this issue.

... but export can waste a lot of time and bandwidth if your project is large ;-)

You can also set your webserver to ignore the .svn folders.

I'd recommend that you look into SVN export command

As others said, set up a repository for your code using a source control system, any one will do. Have a release branch and when updating it, make sure the server grabs it. The updating process can be automated using a check-in hook of some sorts.

I've been using Tortoise SVN - really easy to use,

http://tortoisesvn.tigris.org/

Been using SCP... it's a great tool for copying files between machines... but it's a shitty tool for prod deployment... capistrano ftw!

Use SVN and setup a postcommit thingy that copies the new files to your public directory whenever you do an update.

You'll have a problem if updates involve more than code changes. Any database scripts or configuration changes will be left undone, potentially breaking the site.

You may choose to use a tool to run your update scripts automatically and create configuration files correctly, but you'll want to think it through in advance. A particular problem I have is that certain sensitive passwords should not exist anywhere but in the locations they are required, so even pre-created configuration files still require me to manually enter the password once installed on the server.

I am hugely surprised no one suggested creating packages (rpm, deb etc.) and then deploying it on prod boxes

> svn commit

> svn update

Upload? localhost? pfff. I hack right on the production server!

tar, scp, ssh, (un)tar (whaddya mean it's not a process? it's got all of four steps in it! ^^)

Reply on news.ycombinator.com