Since independence, and particularly recently, Russians (who once made up over half of Latvia's population) have left, and in many cases not been welcome to stay. One of the major tensions in Latvian politics today is the status of Russian Latvians.
With that backdrop (and given the other factors, such as migration to the rest of the EU) it isn't really that surprising that population levels are returning to that of the 60's.
The jump in the graph happened when the most recent economic crisis hit Baltic countries and drove their GDP roughly 20% down in a year.
If you read russian, here is a series of articles with advice on various emigration destinations. It's an interesting view into the current mindset there. http://www.kommersant.ru/doc/2132694