Searching for "fonts ligatures" will give you more reading material.

>I guess this is probably to improve readability but it doesn't do this for other pairs of repeated letters.

Ligatures are mostly a style thing from the days of handwriting that got brought forward into print.

>Is there some pattern matching for sequence of characters ?

Yes, combinations are pre-defined in the font and picked up the renderer. In this particular case, LibreBaskerville defines a combination of two f's to render the first f in a slightly smaller size.