The algorithm used is fairly complicated, being both preferential and proportional. (The lower house is preferential but not proportional).

Here is a nifty visualization of the senate vote flows in NSW: http://www.grwpub.info/senate/nsw.svg.

Essentially you need a certain number of votes to cross the line and win a seat. After winning the seat, those votes are subtracted from the party. Eventually when no parties have enough votes, the lowest voted party is eliminated and its votes are redistributed by preference.