You're completely ignoring the time dilation effects of relativity. From the perspective of a fast moving spaceship, the distance between stars contracts, and from the perspective of an outside observer, the time experienced in the spaceship slows down. This is why I believe that interstellar travel will be feasible, although creating any sort of empire will be impossible. Here's a quote from the Wikipedia article on time dilation:

"Theoretically, time dilation would make it possible for passengers in a fast-moving vehicle to advance further into the future in a short period of their own time. For sufficiently high speeds, the effect is dramatic.[2] For example, one year of travel might correspond to ten years on Earth. Indeed, a constant 1 g acceleration would permit humans to travel through the entire known Universe in one human lifetime.[12] Space travelers could then return to Earth billions of years in the future. A scenario based on this idea was presented in the novel Planet of the Apes by Pierre Boulle, and the Orion Project has been an attempt toward this idea."

https://en.wikipedia.org/wiki/Time_dilation#Velocity_time_di...