It's actually even worse than that. Because of the accelerating expansion of the universe, over time the part of the universe that we can observe will get smaller, allowing us to see less and less of it. Eventually, all that we'll be able to see is our own local group of galaxies, where gravitational attraction will win out over the universe's expansion. However, this won't really be a problem for a few billion years.
Relevant Kurzgesagt video: https://www.youtube.com/watch?v=uzkD5SeuwzM