Lithuanian is closest to the original proto-indo-european language keeping more of it's features than any of the hundreds of other languages that spawned out of it.

It's hard to say any language is "oldest" because languages are alive and the people that speak them change constantly. The pieces that make up the language though are indeed very old regardless of the fact that "Lithuanian" hasn't existed nearly as long.

>Among Indo-European languages, Lithuanian is extraordinarily conservative, retaining many archaic features otherwise found only in ancient languages such as Sanskrit[5] or Ancient Greek. For this reason, it is one of the most important sources in the reconstruction of the Proto-Indo-European language despite its late attestation (with the earliest texts dating only to c. 1500 AD). The phonology and especially the nominal morphology of Lithuanian is almost certainly the most conservative of any living Indo-European language,[4][6] although its verbal morphology is less conservative and may be exceeded by the conservatism of Modern Greek verbs, which maintain a number of archaic features lacking in Lithuanian, such as the synthetic aorist and mediopassive forms.