Some airports have good connections. In the last month I've been to Berlin and Brussels, and in each case my train journey into the city was about 15-20 minutes (caveat: I happened to be going to the right side of Berlin for the airport, and the Brussels train, while central, was _bizarrely_ expensive).
Of course, some airports, not so much. Grumble mutter Dublin (there is some hope of a rail line, or possibly _two_ rail lines, in about 2040, but until then it's a choice of painfully slow standard buses (1 hour into city), or expensive unreliable express buses (25 mins into city, if they show up)).
But for many routes, really security, and the sheer poor layout of the airport, is the big slowdown. My favourite for this is London City; the (small, weird) plane lands, you walk out a door, and you are at a DLR stop.
Boarding also always takes far longer than you'd imagine it should, mostly due to people being people. In principle you could board an airliner in a couple of minutes, but only with perfect behaviour from all passengers, so good luck with that.