Not OP, but the ATTiny is pretty easy to work with because it doesn't need much in the way of external parts. You need an AVR programmer, but you can use an Arduino flashed with the AVR programmer sketch or buy a cheap tinyISP for ~$10. I recommend investing in a programmer but do NOT get a usbASP (usually the cheapest around $5 for clones), they suck. Spring for the $10 clone tinyISP's or one of the legit Adafruit/Arduino/Sparkfun ones.
To get started with e.g. the tiny85 you just lookup the pinouts and connect the programmer, then change the settings in the Arduino IDE to use the programmer instead of serial. If you wanna get more advanced you can upgrade to ATMega's, but those really need an external crystal so it's more parts, but not that bad.
Final step is to stop using Arduino and move to using the native C SDK or even assembly. Make Magazine has a book on programming AVR's, I highly recommend it.
This article is a good starting point that walks you through how to duplicate an Arduino Uno on a breadboard. It's a bit old though so make sure to do some googling for more.