It sounds like you bombed the interview because your answer was not the correct answer for the domain.

For games, it absolutely does matter that only some appends trigger latency, because that causes stuttering in the game play. A linked list may be slower in most use cases...but the performance cost is fixed and can be easily designed around.