1) Theming of form controls to make them look like native widgets.
2) Clipboard.
3) Printing.
4) Fonts and font rendering.
5) General "now it's time to repaint" machinery.
This is in no way an exhaustive list. In practice what you have to do for headless is create the relevant GUI bits anyway, but not actually show them on screen, then deal with whatever quirks your GUI library has when its bits are not shown.
One slight complication of headless has been wiring up the headless "widgets" (in Firefox, we refer to most of the platform specific code as widgets e.g. there's gtk widgets, cocoa widgets, windows widgets). Usually the widget type is defined statically at build time, but in the case of headless we wanted Firefox to either use the headless widgets or the platform specific widgets at runtime. Luckily, some of the work on multi-process Firefox work also added another type of widget and made it much easier to support multiple types of widgets.
Overall, the hardest part has been trying to replicate all the events that would normally be triggered by the platform gui code. I've also found that these events and order can vary per platform which makes it hard to do in a non-platform specific way. We're still working out some issues here.