To get started with e.g. the tiny85 you just lookup the pinouts and connect the programmer, then change the settings in the Arduino IDE to use the programmer instead of serial. If you wanna get more advanced you can upgrade to ATMega's, but those really need an external crystal so it's more parts, but not that bad.
Final step is to stop using Arduino and move to using the native C SDK or even assembly. Make Magazine has a book on programming AVR's, I highly recommend it.
This article is a good starting point that walks you through how to duplicate an Arduino Uno on a breadboard. It's a bit old though so make sure to do some googling for more.
For me it made more sense, because using just 3 x AA I get about 5V so I don't have to use a voltage regulator (less power used).
That way I can also use the attiny's internal voltage reference to measure remaining battery life without any additional circuitry.
Plus the smaller package, no annoying red lights everywhere, etc.
> I would like to the next level and use ATTiny's directly but I have no real electronics experience, how did you get started ?
I started with zero experience as well.
I bought an Arduino, did the LED example and put it away for a year.
Then I found a 37 sensor kit[1], together with a breadboard[2] and some wires[3] and this wonderful wiki[4] and... everything was different, because I was doing real practical (albeit simple) things with it.
The rest was just experimentation and reading about it, which was easy once I've found my "spark".
See @gh02t's answer for the attiny part, I went the lowest friction way with using an existing Arduino to program the attiny84.
[1] https://www.amazon.com/Elegoo-Sensor-Module-Arduino-MEGA/dp/...
[2] http://eud.dx.com/product/solderless-breadboard-with-400-tie...
[3] http://eud.dx.com/product/chuangzhuo-male-to-male-male-to-fe...