Xlib is actually somewhat easier to use than xcb. Xcb however has better support for asynchronous operation. In case of xterm this wouldn't help latency much but would greatly improve throughput.