Now you have a safety net, and you can go nuts with whatever hare-brained schemes you happen to find in a how-to. :) Both Slicehost and Linode have excellent libraries of articles for setting up typical services, although neither one give ongoing maintenance (backups, patching, monitoring, log reviews, etc) nearly enough attention.
If you really want a learning experience, find an old machine, and spend some time with "Linux From Scratch": http://www.linuxfromscratch.org/ . You'll learn an enormous amount about how the whole system fits together by going through that exercise. (I did that with a few machines back in the day before there was something like LFS to point people to; having a roadmap like that would have sped the learning process up significantly.)
Review back the steps, optimized them if necessary. You will find them extremely handy the next time you setup a new box.
The walkthroughs cover a variety of distros including Ubuntu, Debian, and CentOS and can be quite useful regardless of where your VPS is hosted. The articles emphasize good security practices as well which can be helpful for a beginner to understand.
The other thing to do is dig into the docs for anything specific you plan to work on. I tend to look for tools with good documentation before getting started. I've always been impressed with Apache HTTP Server's documentation (http://httpd.apache.org/) and have learned a lot by referencing it when I've had issues.
Whenever you have questions, ask friendly geeks for help. I always do best with real-life assistance, but IRC is a good way to go as well.
1. Go get Nemeth's Linux Sysadmin Handbook. Read it and learn the tricks of the trade.
2. I cannot stress this enough, get a desktop with the exact Linux distro you put up. You never know what you'll learn just working in it day by day.
3. Pick up Sobell's book on the Linux shells. Running a VPS means learning the in's and out's of the shell.
4. Learn about the Apache server if you are using it (or another web server), everything you can get your hands on. Get to know how and at what access level is it running. What its config files are, what tweak goes in it. If you're an avid reader, go get the Apache Cookbook.
5. Make, make, make backups along checkpoints.
6. Don't fear the IRC. Sometimes even just listening in on some channels like where Slicehost is discussed, you'll pick up a lot.
7. Use 'info' where available. And read up on bugtraq if you have the time.
8. Tweak/tune one thing at a time. Starting with the things you know best. Take a backup at each step.
The other thing you'll want to do is understand why things work, not just be a "configuration file programmer".
This sounds borderline zealous, but I recommend installing Gentoo or Arch Linux as your first VPS system if you're just starting out on Linux; either that, or start off with NetBSD if you're willing to go down that route as well. The default install options for each of those distros/OS's are very minimal, and so you're forced to think about exactly what you need to install in order to get a working system.
I second bediger: be sure to investigate _why_ things work, and not _how_ to get it to work. The latter is very important for getting things done, but the former is what trains you in diagnosing obscure system issues.
The problem with Linux, though, is that there are several main families, and even distros in the same family don't really adhere to any specific standard. Some might use syslog, others syslog-ng. Some sendmail, others postfix. It's kind of a mess.
Familiarizing yourself with a few flavors (and maybe FreeBSD or OpenBSD for good measure) on a machine you force yourself to use on occasion, you will find your comfort level for basic UNIX power-user and sysadmin stuff will grow.
http://www.freesoftwaremagazine.com/articles/hardening_linux
I really like OpenBSD (and the minimalism and excellent documentation are real advantages for learning systems administration), but since you're using prgmr.com, you would probably be better off using a distro they provide* - Debian, perhaps. While there are probably faster ways to learn sysadmin skills, I learned what I know by just dealing with all the issues that have come up over a decade. (And sooner or later, knowing ed will pay off.)
* Though I've found experience with BSD to transfer better to Unix-in-general than Linux. This is probably an even mix of cultural factors and history - BSD is pretty old-school.
A lot of it depends on what you are trying to 'serve' :)
EDIT: Make sure you spend some time reading about security issues (iptables and such). It's easy to miss, but certainly as important as any other sys-admin task
Jump feet first and use Linux as your main OS. If you are doing development work, it can easily surpass Windows, and is comparable (or superior) to OSX as well. Of course, this doesn't apply if you are doing graphic design stuff, however, Windows can live inside a VM for that sort of stuff. If you are doing web dev, you'll need a VM for Vista/Win7 and for XP anyways.
I wouldn't have recommend it about a year ago but nowadays it's caught up with others quite nicely feature wise, is very fast, supports snapshots and USB, and is completely free.
If you're only ever on a Mac, I vote VMware Fusion, but if you're going to change vm host environments, I'd suggest VirtualBox.
Oh, and above and beyond, don't even think about trying to mess with things that will require a reboot quite yet. Wait until you are much more capable. I've been using Linux for 15 years and OpenBSD for 8, and I still get cold palms when working remotely on my server if it requires a reboot!
Now as soon as you have users or a business, etc... things change quickly. Your goal will change from learning to maintaining or stability. That's why multiple environments are so common. Most people in IT are not learning they are optimizing, maintaining and protecting. So you have to understand when viewing most comments and information on the internet that you have a different goal (learning) than most of the IT industry.
Almost all the most useful ideas that I learned about managing a server, I learned due to an experience. Opposed to just reading about it on serverfault or superuser :)
prgmr.com is out of space though. Does anyone know of anything else like this out there?
For learning, I'd start with the 256 or lowest RAM setup and plan to upgrade later if you get traffic.
Some of the most popular VPS hosts are linode and slicehost but there are many.
It's cheap and easy to upgrade your laptop from 2GB to 4GB.
It can be expensive to upgrade your laptop from 4GB to 8GB, no matter how cheap the RAM itself is, because you might have to buy a new laptop to accomodate.
It's a much more expensive (and logistically difficult) task to roll out that kind of upgrade across a large hosting environment with hardware purchased over multiple deployments.
http://blog.linode.com/2010/06/16/linode-turns-7-big-ram-inc...
None the less, new servers with lots of ram aren't that expensive. Registered ecc ddr3 is about $130 for each 4GiB stick, give or take, and that comprises about half the cost of a server with 8 cores and 32giB ram. (on the G34 platform, that's a single socket server)
This depends a lot on the stack you're using and what software your site depends on.
You will also need a bunch of O'Reilly books.
Being able to use the proper tool (control panel or straight command line) is best. I'd found that using a tool to do X would let me then go look at the config files before and after X to see what/how the tool was doing - that was a great learning exercise for me. Might not be for others though.
also, google is your friend for knowing what but often not why.