It's outlined here: http://www.x.org/wiki/guide/concepts/#xisclientserver . In a nutshell, X "serves" the application to the (in- and output) devices.